How the test-and-learn process supports business innovation

What are common challenges in determining cause-and-effect relationships?

How is the test-and-learn process different from other analytic methodologies?

The test-and-learn process involves implementing a change for select test groups and withholding that change for matched control groups. For example, a financial institution could send a credit card promotion to select customers only, or a retailer could implement a new store layout in some locations, but not all.  

Test versus control experimentation is the analytic gold standard for determining casual relationships. Other analytic methods (e.g., regression analyses) are helpful hypothesis generators, but only a business experiment that observes actual performance can isolate the true causes and effects of an initiative. 

In an uncontrolled environment, there are unlimited external factors that can affect performance, making it extremely difficult to isolate the true incremental impact of an action. An internal Mastercard analysis found that 45% of campaigns have natural test vs. control bias, meaning there is great room for improvement in analysis structuring.

Below is a walkthrough of the test-and-learn process, from hypothesis to execution.

Step 1: Generate hypothesis 

The process always begins with creating a testable hypothesis.

Example: A retailer believes that a change to store layout will drive customers to visit more frequently, thus increasing sales.

Important considerations:

• A hypothesis must be testable and have realistic success criteria that are defined and measurable.
• It should be possible to pass or fail the test based on the measured goals of the hypothesis.
• Stakeholders should align internally on when a test is necessary and when it is unnecessary.

What a company must ask themselves before conducting a business experiment:

• Does the experiment have a clear purpose?
• Is the experiment doable?
• Have stakeholders committed to abide by the results?
• How can we ensure reliable results?
• How do we get the most value out of the experiment?

Step 2: Design and implement test

This step ensures that the test is properly designed and variables are identified in advance. 

Example: The retailer identifies highly similar “control” sites where the store layout change didn’t occur and uses them as a performance baseline.

Important considerations:

• Defining test variables and success criteria ensures conclusive learning and positions the business to be able to move forward with a decision. The test should be set up to address the business goals and minimize bias. 
• Appropriate definition of the test’s size and scope can generate more accurate results. Businesses should make sure they have:
  • A large enough pool to form robust test and control groups
  • An appropriate test duration to overcome random noise
  • A finalized test schedule to avoid overlap with other initiatives
     

Step 3: Measure impact across all channels

Rigorous tracking and monitoring of key performance indicators allows businesses to feel confident that impact is directly related to their initiative and not simply the result of random chance. 

Example: Using test versus control methodology, the retailer measures the true incremental sales impact driven by their store layout change. 

Important considerations:

• True impact is quantified by evaluating the performance before and after an initiative for both the test and control groups, then calculating what change in performance can be attributed to the initiative. 
• Results should be analyzed to check for a halo or cannibalization effect on other business areas.
• Thorough analysis can identify the most effective aspects of the initiative and the types of customers or markets that respond best to it.

Step 4: Roll out targeted initiative

Once an initiative is determined to be successful, a business can roll it out to the sites or customers where it is expected to perform best.

Example: The retailer implements the layout changes in the stores expected to perform most positively. 

Important considerations:

• Making a go/no-go decision is relatively simple for resounding successes or definitive failures. For tests that don’t fall nicely in either category, identifying targeted rollouts can unlock a lot of value.
• Organizations can optimize performance by modifying the initiative based on its most effective aspects and rolling it out in a targeted manner to a group of customers or locations predicted to respond best. 
• Effectively communicating the results of the test can help to align stakeholders on the subsequent rollout.

Step 5: Learn and repeat

Results and learnings from a test shouldn’t just be discarded once it has been run. Ideally, they should be collected so others can refer to them, avoiding repetition, providing future inspiration and contributing to the greater institutionalization of the test-and-learn process within organizations.

Example: Learnings from the retailer’s store layout change test help inform best practices for future layout and merchandising initiatives. 

Important considerations:

• Cataloging test results allows other members of the organization to quickly gain insight into previous tests, preventing repeat testing and wasted resources.
• All measurements hold value. They can unlock new insights beyond the initial testing parameters and be used to form new hypotheses.
• Executives may hold off on rolling out an initiative until more testing is done. The organization can use previous tests to identify the best-performing aspects and isolate those variables in a new test. 

Conclusion

The test-and-learn process helps businesses optimize new strategies, minimize risk and innovate more quickly. By institutionalizing a testing culture, organizations can better understand the causal relationship between business decisions and financial outcomes, enabling them to action on the most effective strategies possible.

To learn how Mastercard Test & Learn® helps companies make more confident business decisions by accurately analyzing and optimizing new ideas and ongoing initiatives, contact us. 

Want to read more about the test-and-learn process? View The essential guide to business experimentation for examples and tips.