Skip to main content
Hero Banner
Business owner using business experimentation tool on tablet

How the test-and-learn process supports business innovation

A walkthrough of the business experimentation process from hypothesis to execution.

By: Sarah Payne

Published: April 19, 2024 | Updated: August 16, 2024

Read time: 6 minutes

Table of Contents

 

Introduction to the test-and-learn process

Data can serve as a window into customer preferences and needs. However, many companies lack the ability to extract meaningful insights from their data at scale. Businesses know they need to make more informed decisions but may not know how. Enter the test-and-learn process.

Explore more

The essential guide to business experimentation
Maximizing promotion strategy with business experimentation
How retailers can optimize that consumer engagement sweet spot

 

What is the test-and-learn process?

Just as a scientist develops hypotheses and collects data to assess the validity of their assumptions, leading companies engage in small-scale “business experiments” to make more informed, data-driven decisions. 

When businesses introduce testing and learning to their strategic decision-making, they can cut through noise and ambiguity to identify and act on the highest ROI initiatives possible. With this rigorous testing method, scientifically sound experiments reveal the overall incremental impact of an initiative, the most effective aspects of that initiative, the types of customers or markets that respond best and how to target rollout across the organization.

Testing ultimately accelerates innovation by enabling businesses to rapidly explore ideas within a controlled environment without the risk of rolling out an ineffective program to the entire organization.

 

Test and learn cycle

 

It can be challenging to identify the true business impact of a new initiative; however, the test-and-learn process helps companies understand what works best and why. Before committing to a full rollout, an intentionally crafted business experiment can reveal:

  • The impact on key performance metrics
  • How to tailor and target an initiative for maximum effectiveness

 

What is the value of the test-and-learn process in business?

Based on a Mastercard Test & Learn® client survey, 44% of business initiatives don’t break even. Used by many leading brands, in-market testing empowers business leaders to improve the success rate of their initiatives and understand their overall incremental impact. By mitigating certain risks, companies can feel more confident in their business strategies.

In short, the test-and-learn process helps organizations maximize learnings, minimize risks and stimulate innovation.
 

Test & Learn®
Start testing new ways to engage consumers
Learn more

 

What are common challenges in determining cause-and-effect relationships?

 

How is the test-and-learn process different from other analytic methodologies?

The test-and-learn process involves implementing a change for select test groups and withholding that change for matched control groups. For example, a financial institution could send a credit card promotion to select customers only, or a retailer could implement a new store layout in some locations, but not all.  

Test versus control experimentation is the analytic gold standard for determining casual relationships. Other analytic methods (e.g., regression analyses) are helpful hypothesis generators, but only a business experiment that observes actual performance can isolate the true causes and effects of an initiative. 

In an uncontrolled environment, there are unlimited external factors that can affect performance, making it extremely difficult to isolate the true incremental impact of an action. An internal Mastercard analysis found that 45% of campaigns have natural test vs. control bias, meaning there is great room for improvement in analysis structuring.

 

test-and-learn illustration

 

Below is a walkthrough of the test-and-learn process, from hypothesis to execution.

 

Step 1: Generate hypothesis 

The process always begins with creating a testable hypothesis.

Example: A retailer believes that a change to store layout will drive customers to visit more frequently, thus increasing sales.

Important considerations:

  • A hypothesis must be testable and have realistic success criteria that are defined and measurable.
  • It should be possible to pass or fail the test based on the measured goals of the hypothesis.
  • Stakeholders should align internally on when a test is necessary and when it is unnecessary.

What a company must ask themselves before conducting a business experiment:

  • Does the experiment have a clear purpose?
  • Is the experiment doable?
  • Have stakeholders committed to abide by the results?
  • How can we ensure reliable results?
  • How do we get the most value out of the experiment?

 

Step 2: Design and implement test

This step ensures that the test is properly designed and variables are identified in advance. 

Example: The retailer identifies highly similar “control” sites where the store layout change didn’t occur and uses them as a performance baseline.

Important considerations:

  • Defining test variables and success criteria ensures conclusive learning and positions the business to be able to move forward with a decision. The test should be set up to address the business goals and minimize bias. 
  • Appropriate definition of the test’s size and scope can generate more accurate results. Businesses should make sure they have:
    • A large enough pool to form robust test and control groups
    • An appropriate test duration to overcome random noise
    • A finalized test schedule to avoid overlap with other initiatives
       
In a business experiment, an initiative is implemented in some sites (or with some customers) and not others.

 

Step 3: Measure impact across all channels

Rigorous tracking and monitoring of key performance indicators allows businesses to feel confident that impact is directly related to their initiative and not simply the result of random chance. 

Example: Using test versus control methodology, the retailer measures the true incremental sales impact driven by their store layout change. 

Important considerations:

  • True impact is quantified by evaluating the performance before and after an initiative for both the test and control groups, then calculating what change in performance can be attributed to the initiative. 
  • Results should be analyzed to check for a halo or cannibalization effect on other business areas.
  • Thorough analysis can identify the most effective aspects of the initiative and the types of customers or markets that respond best to it.
Category impact graph
Store age graph
overall impact
Testing answers two key questions: "Did my initiative work?" and "Why or why not?"

 

Step 4: Roll out targeted initiative

Once an initiative is determined to be successful, a business can roll it out to the sites or customers where it is expected to perform best.

Example: The retailer implements the layout changes in the stores expected to perform most positively. 

Important considerations:

  • Making a go/no-go decision is relatively simple for resounding successes or definitive failures. For tests that don’t fall nicely in either category, identifying targeted rollouts can unlock a lot of value.
  • Organizations can optimize performance by modifying the initiative based on its most effective aspects and rolling it out in a targeted manner to a group of customers or locations predicted to respond best. 
  • Effectively communicating the results of the test can help to align stakeholders on the subsequent rollout.

 

Project benefit graph

 

Step 5: Learn and repeat

Results and learnings from a test shouldn’t just be discarded once it has been run. Ideally, they should be collected so others can refer to them, avoiding repetition, providing future inspiration and contributing to the greater institutionalization of the test-and-learn process within organizations.

Example: Learnings from the retailer’s store layout change test help inform best practices for future layout and merchandising initiatives. 

Important considerations:

  • Cataloging test results allows other members of the organization to quickly gain insight into previous tests, preventing repeat testing and wasted resources.
  • All measurements hold value. They can unlock new insights beyond the initial testing parameters and be used to form new hypotheses.
  • Executives may hold off on rolling out an initiative until more testing is done. The organization can use previous tests to identify the best-performing aspects and isolate those variables in a new test. 

 

Conclusion

The test-and-learn process helps businesses optimize new strategies, minimize risk and innovate more quickly. By institutionalizing a testing culture, organizations can better understand the causal relationship between business decisions and financial outcomes, enabling them to action on the most effective strategies possible.

To learn how Mastercard Test & Learn® helps companies make more confident business decisions by accurately analyzing and optimizing new ideas and ongoing initiatives, contact us

Want to read more about the test-and-learn process? View The essential guide to business experimentation for examples and tips.
 

Related resources

The-Essential-Guide-to-Business-Experimentation_Teaser
Blog
The essential guide to business experimentation

Business experimentation can help organizations evaluate new initiatives. View this guide to learn more about the testing process, challenges, examples and tips.

T&L Vid
Video
Navigating Inflation with Test & Learn 

Explore how Test & Learn can help retailers navigate inflation.

eclipse tile image
Blog
Eventful Economy: Total eclipse of the economy

The 2024 total solar eclipse was a highly anticipated event in the United States, with millions of Americans across the country traveling to witness the celestial phenomenon. How did this impact the economy?