Assessing the analytics and system approach taken involves collaborating as an analytics team to determine whether the results from data exploration or data analysis are helping to answer the business question. Assessing the analytics approach is performed iteratively with Explore Data and Analyze Data.

When issues arise with data sourcing or with the results of data analysis, the approach to analytics adapts. For example, when data exploration uncovers issues with data quality or determines the wrong data is being collected, or data gaps are an issue, there may be a need for adjustments to be made to how and where the data is being collected. If the results of data exploration are acceptable, it is still possible the results from data analysis will fail to answer the questions being asked. The results from data analysis may not produce results that help meet the objectives of the initiative.

In these scenarios, data exploration and data analysis tasks are repeated. Iteration occurs between the data exploration and data analysis tasks until the data scientist is comfortable with the data sources being used. Their assessment is based on the quality of data being obtained and its value toward answering the research questions.

When assessing the analytics and system approach taken, business analysis professionals require basic skills in statistics and a basic understanding of data science tools and technologies. They should possess sufficient business acumen to provide context to the data analysis. Business analysis professionals answer questions the data scientist may pose related to the business. Adaptability is necessary to adjust the analysis approach as more data is uncovered, new insights learned, or different levels of stakeholders are involved. Trustworthiness is important as in some industries, having access to certain types of data comes with a great deal of responsibility, often with legal implications. It is important to know what acceptable data use is and what it is not, and what can be accessed or viewed and what cannot.

Real-World Example of Analytics Challenges and Course Correction In highly competitive markets, organizations struggle to get the right message to the customers, at the right time. To achieve this, marketing teams use “hyper- personalization” to target their messaging to their customers. Whether a particular campaign will have a significant impact on sales is a question of concern for most marketing teams. This is a research problem. For example, what is the likely outcome if a campaign is launched and which potential customers are likely to purchase. This is an example of a question suited for prescriptive analytics. Consider an organization launching a campaign to offer a 10% discount on a digital product. The organization also wants to reduce the cost of campaigning by limiting the number of customers they want to target. The organization collected data on 50,000 customers and leads from their CRM system (often referred to as population or population set), and were trying to decide what the campaign will be, in terms of potential customers who will buy the product. The data included variables such as age, geography, gender, different product features, existing customer or not, and has the customer purchased the product in the past or not. The analytical approach, broadly, was as follows: People who have already purchased the product carry certain attributes and people who have similar attributes will likely purchase the product, given a discount. Out of the 50,000 customer records, a subset of 30,000 customer records was chosen to train the model (for example, in simple terms, training means to deduce the mathematical construct that predicts the outcome using a subset of data). This training data of 30,000 records included customers who had already purchased the product and some who have not. Remaining data within the population set can be used for testing the prediction accuracy by comparing the prediction from the mathematical model versus the testing data (for example, the remaining data) where we know whether the customer has purchased the product or not. If the test data accuracy is high, the model can be applied to successively larger or different sets of data segments to predict who are likely to purchase the product and the campaign can be exposed to those customers only. An algorithmic model (for example, logistic regression) was considered and although the training performance was considerably high, the testing performance was found to be low. The data scientist suggested there is a possibility the variable considered may not be a true predictor for the given problem. For example, there are two groups, existing customers and leads, who will have different purchasing behaviours even though the values of the other variables are similar. Additionally, there may be other variables missing from analysis such as income or level of education, which are influencing one or more variables that are part of the analysis and the missing variables may be the true predictors. In statistical terms, these are called confounding variables. The analytics initiative now has a choice: Obtain more relevant data (for example, income and education) with the additional cost of surveys as data may not be available readily. Conduct randomized simulation or A/B testing by pursuing purely random customers to study the effectiveness of the campaign but with a higher cost of designing a new experiment and analytical approach. Change the analytical approach and models that are better suited where confounding variables are involved such as propensity score matching or random forest methods. For all these scenarios, the analyst plays a significant role in evaluating and assessing cost-benefit, feasibility, and business impact. This deliberation requires multiple stakeholder discussions and consensus. These are typical business analysis skills in addition to a functional knowledge of analytical methods and concepts.