The essential guide to business experimentation

What KPI’s are involved in business experimentation? 

Business experiments can measure a variety of KPIs, depending on the initiative and the type of business running the experiment. For example, a restaurant that is adding a new drive-thru may focus on measuring the impact on transactions and sales. A loyalty program manager, on the other hand, may be more interested in customer lifetime value.

What key terms are used in business experimentation?

• Bias - External factors can lead to distorted results and inaccurate conclusions.
• Control group - Locations or customers that were intentionally excluded from the test treatment. This will serve as a baseline to determine the efficacy of the test.

Key steps in a business experiment

There are five steps in the business experimentation process:

Step 1: Create a hypothesis

Business hypotheses can be centered around solving a problem or capitalizing on an emerging opportunity. To create the most effective hypotheses, organizations should develop a deep understanding of the latest industry trends and customer purchasing patterns.

Step 2: Design and implement a test

To accurately measure the impact of any business action, organizations must first minimize bias in the target population and then isolate the test segment from outside influences. This can be achieved with rigorous test and control sizing and selection using advanced simulations.

Step 3: Measure the impact across all channels

Organizations should then measure the performance before and after an initiative across all channels for both the test and control groups. With a well-designed experiment, any differences in performance between the “test” and “control” groups can be directly attributed to the initiative.

Step 4: Roll out your targeted initiative

By drilling down further and scanning hundreds of attributes, organizations can identify the strongest drivers of performance. These drivers can be combined into a predictive model, which can show how to target the roll out to maximize the benefit.

Step 5: Learn and repeat

Organizations should use learnings from current business experiments to inspire new ideas and hypotheses for future tests.

Blockers and challenges

There are a few challenges that companies face in the business experimentation process:

Let’s walk through a few different business experimentation examples across industries that showcase how leading brands are overcoming these challenges.

Examples of successful business experiments

By experiment:

• Analyzing NFL game day ads
• Measuring loyalty program effectiveness
• Improving operational efficiencies
• Understanding if retention offers work
• Calculating ROI of a restaurant promotion
• Evaluating the impact of remodeling stores

By industry:

• Consumer packaged goods
• Convenience retailers
• Banking
• Telecommunications
• Quick service restaurants (QSRs)
• Apparel retailers

Business experimentation for consumer packaged goods (CPG) companies

A beverage brand in North America worked with Mastercard to analyze the effectiveness of its NFL game day television ads. Although the most accurate way to measure an initiative’s impact is with a designed test and control group, the NFL game day ads were rolled out to the entire market and there was no control for comparison.

By using Test & Learn® with SpendingPulse™ Modeling, the CPG brand was able to analyze the effectiveness of their ads. Rather than a test vs. control approach, this methodology predicted the brand’s expected performance in the absence of the ads by using machine learning, proprietary Mastercard insights and correlated datasets. The analysis showed which products benefited the most from the ads and what timeframes worked best. With these insights, the brand developed more effective marketing investments.

Business experimentation for convenience retailers

A convenience retailer introduced SessionM to power their loyalty program. However, they struggled to test the loyalty program’s true performance due to their limited control-matching capabilities.

By using consumer permissioned first-party data collected on the SessionM platform, the retailer was able to create a well-designed experiment with Mastercard Test & Learn®, reducing test and control bias from 22% to under 6%. The loyalty program test found that the retailer’s SessionM loyalty offers drove an 8.8% lift in incremental sales.

To further enhance their SessionM-powered loyalty program, the retailer is now using Test & Learn® to consistently optimize outreach tactics, target promotions to high-potential customer segments and send win-back offers to lapsed customers.

Business experimentation for banks

A global retail bank wanted to install Smart ATMs to save costs on labor and transaction processing, but some executives worried the change would not be well-received by certain customer groups.

Test & Learn® helped the retail bank use best practices like test settings with analysis period and outlier removal. It also used proprietary control-matching algorithms to remove bias, matching control branches to test locations.

The tool identified that branches located in areas with younger populations and less competition, and those located in areas with larger households, responded best to the change. The bank estimated a $17 million annual profit improvement using Test & Learn® recommendations to target the rollout of the Smart ATMs.

Business experimentation for telecommunications

A US telecommunications provider was struggling to effectively measure the impact of retention offers. Some executives believed that retention offers would reduce attrition. However, others argued that some customers were never at risk of leaving and other customers were purely selecting the most attractive offers and lapsing once those ended.

One challenge in this scenario was that there was no initial control group. The provider used Test & Learn® to build synthetic control groups based on customer profiles. Synthetic control methodology is the process of retroactively identifying well-matched control groups in situations where there was no initial randomized control group. As a result, Test & Learn® was able to evaluate the offers and found that the retention offers were effective, leading to a 3.3% decrease in churn.

Business experimentation for quick service restaurants (QSR)

A large QSR chain wanted to understand how well its restaurant promotion strategy worked, as well as what might happen if they invested more in the strategy. There were many outside influences that could affect performance, which made it difficult to isolate what the promotion changed.

Test & Learn® helped design an experiment where some restaurants launched the promotion and others didn’t. Performance was compared between the two, evaluating features like promoted item, discount amount and duration. The restaurant chain found that the promotion led to a 1.3% sales increase per customer and that shorter promotions with larger discounts were most effective.

Maximizing promotion strategy with business experimentation

Overall, today’s consumers are searching for the best deals, so promotions and price cuts are valuable tools to get more business. To learn more about how to maximize your promotion strategy with business experimentation, including an example from US retailer J.Crew, check out this blog.

Business experimentation for apparel retailers

A large apparel retailer was remodeling stores to improve sales of its new athletic apparel line. Before investing across the network, the retailer wanted to understand the incremental impact of the remodel.

Their first analysis showed that the remodeled stores experienced a 3.6% lift in total sales, but this lift wasn’t enough to cover the costs of remodeling the stores. Before deciding to cancel the whole project, the retailer decided to run a more robust analysis using Test & Learn®.

The Test & Learn® analysis looked at hundreds of attributes to find the strongest drivers of performance. For example, the data showed that new light fixtures boosted sales and that stores located in areas with a higher percentage of young adults responded most positively to the remodel.

Instead of viewing the remodel as a go or no-go decision, the retailer used Test & Learn® insights to create a tailored and targeted approach. It drove over $9 million in net present value (NPV) in year one of the remodel program.

3 tips for successful experiments

Tip 1: Explore beyond your hypothesis

It’s important to develop and evaluate a hypothesis as part of your business experimentation process, but as Proctor & Gamble shared at the Bentonville Excellence in Analytics Summit, sometimes the most interesting findings don’t directly relate to your hypothesis. Be on the lookout for learnings!

Tip 2: Remove biases

Many business experiments start off with natural test vs. control bias, but that makes it tough to measure the incremental impact of an initiative. To overcome this, be sure to use precise control matching strategies across all relevant attributes.

Tip 3: Quickly scale

Conducting business experiments can take time and use a lot of resources. For some teams, this means choosing between the number of analyses, depth of insights each produces and speed of getting answers for decision makers. To optimize how you execute your experiments, partner with a business experimentation firm.

Conclusion

Business experimentation is the only way to truly know what will happen when an initiative is deployed. Other analytic methods, such as regression analyses, are helpful when creating hypotheses, but only a business experiment that observes actual performance can tell you the true cause-and-effect impact of an initiative across all channels.