Skip to main content
Hero Banner
The-Essential-Guide-to-Business-Experimentation_Banner

The essential guide to business experimentation

By: Katheryn Stanwick, Sarah Payne and Erica Susi

Published: April 15, 2024 | Updated: August 16, 2024

Read time: 9 minutes

Table of Contents:

 

Introduction

To stay ahead of the competition, your business may want to introduce new initiatives, such as bringing an innovative product to market. Before going any further, you may want to ask a few questions:

  • Will the new product cannibalize sales of existing products or drive truly incremental benefits?
  • How should we price the product to optimize profits and customer loyalty?
  • Which markets will respond best to this new offering?

To answer these questions, you may want to engage in the business experimentation process.

Business experimentation can help organizations in any industry evaluate new initiatives before rolling them out to broader audiences or locations. For a restaurant business, for example, business experiments could include:

  • New operational hours
  • Restaurant remodels
  • Pricing adjustments
  • Employee staffing and training programs
  • New menu items
  • Multimedia campaigns
  • Customer loyalty initiatives

 

Terms and definitions

 

What is experimentation in business?  

Business experimentation refers to the process of testing a new initiative in some locations or with some customers and comparing performance in the “test group” to a well-matched “control group” that does not receive the new initiative.

Performance metrics like sales and profit can be extremely volatile due to factors like weather, changes in socioeconomic conditions and competitor actions. Isolating the impact of a program amidst all of this “noise” is tough, and you might ask yourself: Did my initiative really work, or was it just due to external factors? 

When businesses experiment with test and control groups that are similar in their characteristics or behaviors, they can minimize the influence of factors that they can’t control. Any difference in the experiment’s results is from true cause-and-effect between what they did (for example, a restaurant makeover) and what they measured (for example, a 3% increase in sales). 

 

Why do businesses need to experiment? 

Businesses need to experiment because well-designed business experiments fuel innovation in a cost-effective way. Experiments empower businesses to try new ideas on a small scale before investing in a more resource-intensive rollout.  

 

Who is involved in the business experimentation process? 

The stakeholders involved in the business experimentation process vary depending on the type of initiative. For instance, the marketing department should be heavily involved when testing a new promotional offer, while finance and real estate teams would weigh in more on large-scale store remodels.

 

Which tools should a business use to conduct experiments? 

Organizations can use business experimentation to understand how new initiatives will perform. Experimentation platforms like Mastercard Test & Learn® can help isolate the true incremental impact of an initiative. Remember: Big data can provide insights on historical behaviors, but data alone isn’t enough to fuel innovation.

Test & Learn®
To learn more about business experimentation platforms, please visit our Test & Learn® page.
Learn more

What KPI’s are involved in business experimentation? 

Business experiments can measure a variety of KPIs, depending on the initiative and the type of business running the experiment. For example, a restaurant that is adding a new drive-thru may focus on measuring the impact on transactions and sales. A loyalty program manager, on the other hand, may be more interested in customer lifetime value.

 

What key terms are used in business experimentation?

  • Bias - External factors can lead to distorted results and inaccurate conclusions.
  • Control group - Locations or customers that were intentionally excluded from the test treatment. This will serve as a baseline to determine the efficacy of the test.

 

Key steps in a business experiment

There are five steps in the business experimentation process:

 

Step 1: Create a hypothesis

Business hypotheses can be centered around solving a problem or capitalizing on an emerging opportunity. To create the most effective hypotheses, organizations should develop a deep understanding of the latest industry trends and customer purchasing patterns.

 

Step 2: Design and implement a test

To accurately measure the impact of any business action, organizations must first minimize bias in the target population and then isolate the test segment from outside influences. This can be achieved with rigorous test and control sizing and selection using advanced simulations.

 

Step 3: Measure the impact across all channels

Organizations should then measure the performance before and after an initiative across all channels for both the test and control groups. With a well-designed experiment, any differences in performance between the “test” and “control” groups can be directly attributed to the initiative.

 

Step 4: Roll out your targeted initiative

By drilling down further and scanning hundreds of attributes, organizations can identify the strongest drivers of performance. These drivers can be combined into a predictive model, which can show how to target the roll out to maximize the benefit.

 

Step 5: Learn and repeat

Organizations should use learnings from current business experiments to inspire new ideas and hypotheses for future tests.

 

Blockers and challenges

There are a few challenges that companies face in the business experimentation process:

Let’s walk through a few different business experimentation examples across industries that showcase how leading brands are overcoming these challenges.

 

Examples of successful business experiments

By experiment:

By industry:

 

Business experimentation for consumer packaged goods (CPG) companies

A beverage brand in North America worked with Mastercard to analyze the effectiveness of its NFL game day television ads. Although the most accurate way to measure an initiative’s impact is with a designed test and control group, the NFL game day ads were rolled out to the entire market and there was no control for comparison.

By using Test & Learn® with SpendingPulse™ Modeling, the CPG brand was able to analyze the effectiveness of their ads. Rather than a test vs. control approach, this methodology predicted the brand’s expected performance in the absence of the ads by using machine learning, proprietary Mastercard insights and correlated datasets. The analysis showed which products benefited the most from the ads and what timeframes worked best. With these insights, the brand developed more effective marketing investments.

 

Business experimentation for convenience retailers

Convenience retailer

A convenience retailer introduced SessionM to power their loyalty program. However, they struggled to test the loyalty program’s true performance due to their limited control-matching capabilities.

By using consumer permissioned first-party data collected on the SessionM platform, the retailer was able to create a well-designed experiment with Mastercard Test & Learn®, reducing test and control bias from 22% to under 6%. The loyalty program test found that the retailer’s SessionM loyalty offers drove an 8.8% lift in incremental sales.

To further enhance their SessionM-powered loyalty program, the retailer is now using Test & Learn® to consistently optimize outreach tactics, target promotions to high-potential customer segments and send win-back offers to lapsed customers.

 

Business experimentation for banks

Banks

A global retail bank wanted to install Smart ATMs to save costs on labor and transaction processing, but some executives worried the change would not be well-received by certain customer groups.

Test & Learn® helped the retail bank use best practices like test settings with analysis period and outlier removal. It also used proprietary control-matching algorithms to remove bias, matching control branches to test locations.

The tool identified that branches located in areas with younger populations and less competition, and those located in areas with larger households, responded best to the change. The bank estimated a $17 million annual profit improvement using Test & Learn® recommendations to target the rollout of the Smart ATMs.

 

Business experimentation for telecommunications

Telecommunications

A US telecommunications provider was struggling to effectively measure the impact of retention offers. Some executives believed that retention offers would reduce attrition. However, others argued that some customers were never at risk of leaving and other customers were purely selecting the most attractive offers and lapsing once those ended.

One challenge in this scenario was that there was no initial control group. The provider used Test & Learn® to build synthetic control groups based on customer profiles. Synthetic control methodology is the process of retroactively identifying well-matched control groups in situations where there was no initial randomized control group. As a result, Test & Learn® was able to evaluate the offers and found that the retention offers were effective, leading to a 3.3% decrease in churn.

 

Business experimentation for quick service restaurants (QSR)

A large QSR chain wanted to understand how well its restaurant promotion strategy worked, as well as what might happen if they invested more in the strategy. There were many outside influences that could affect performance, which made it difficult to isolate what the promotion changed.

Test & Learn® helped design an experiment where some restaurants launched the promotion and others didn’t. Performance was compared between the two, evaluating features like promoted item, discount amount and duration. The restaurant chain found that the promotion led to a 1.3% sales increase per customer and that shorter promotions with larger discounts were most effective.

Maximizing promotion strategy with business experimentation

Overall, today’s consumers are searching for the best deals, so promotions and price cuts are valuable tools to get more business. To learn more about how to maximize your promotion strategy with business experimentation, including an example from US retailer J.Crew, check out this blog.

Business experimentation for apparel retailers

A large apparel retailer was remodeling stores to improve sales of its new athletic apparel line. Before investing across the network, the retailer wanted to understand the incremental impact of the remodel.

Their first analysis showed that the remodeled stores experienced a 3.6% lift in total sales, but this lift wasn’t enough to cover the costs of remodeling the stores. Before deciding to cancel the whole project, the retailer decided to run a more robust analysis using Test & Learn®.

The Test & Learn® analysis looked at hundreds of attributes to find the strongest drivers of performance. For example, the data showed that new light fixtures boosted sales and that stores located in areas with a higher percentage of young adults responded most positively to the remodel.

Instead of viewing the remodel as a go or no-go decision, the retailer used Test & Learn® insights to create a tailored and targeted approach. It drove over $9 million in net present value (NPV) in year one of the remodel program.

 

3 tips for successful experiments

 

Tip 1: Explore beyond your hypothesis

It’s important to develop and evaluate a hypothesis as part of your business experimentation process, but as Proctor & Gamble shared at the Bentonville Excellence in Analytics Summit, sometimes the most interesting findings don’t directly relate to your hypothesis. Be on the lookout for learnings!

 

Tip 2: Remove biases

Many business experiments start off with natural test vs. control bias, but that makes it tough to measure the incremental impact of an initiative. To overcome this, be sure to use precise control matching strategies across all relevant attributes.

 

Tip 3: Quickly scale

Conducting business experiments can take time and use a lot of resources. For some teams, this means choosing between the number of analyses, depth of insights each produces and speed of getting answers for decision makers. To optimize how you execute your experiments, partner with a business experimentation firm.

 

Conclusion

Business experimentation is the only way to truly know what will happen when an initiative is deployed. Other analytic methods, such as regression analyses, are helpful when creating hypotheses, but only a business experiment that observes actual performance can tell you the true cause-and-effect impact of an initiative across all channels.

Related resources

Retail trends of 2024 teaser
Report
Retail industry trends in 2024

Read about the 16 retail industry trends paving the way for the year of the “empowered consumer” in 2024.

CPG trends in 2024 teaser
Report
Consumer goods industry trends 2024

Consumer goods companies face competing demands from consumers, retailers and a rebalancing economy. What are the consumer goods industry trends for 2024?

e24 tile image
Report
Economic Outlook 2024

Our economic outlook for 2024 includes steady growth, empowered consumers, and macroeconomic risks. See regional breakouts for the US, Canada, China & more.