A/B testing has become a cornerstone of data-driven decision-making in the digital age. It’s a tool that empowers teams to evaluate ideas, combat biases, and uncover user insights. This article will explore the essence of A/B testing, from its foundational principles to practical applications, drawing lessons from industry expertise.

What Is A/B Testing?

• Variant A: We can call this variously: control variant, original variant, unchanged pattern, existing feature, or null hypothesis.
• Variant B: test variant, name_of_test pattern, or reject-null hypothesis

A/B test in the nutshell is an online controlled experiment

A/B test can be A/B/C/D/…./X/Y/Z test

Why Conduct A/B Tests?

A/B testing isn’t just about validating ideas; it’s about empowering teams and reducing the fear of failure. Key benefits include:

Benefits of using A/B test in your project

• Evaluate ideas before launching the feature to 100 percent of your user
• Increase insights about your users — how they engage with the product and react to changes as the product evolves
• Combat bias to create more inclusive products.
• Empower your team so anyone can test an idea regardless of their job title
• Given data insights from past experiments, this will enable better decision-making in the future as the product evolves. It will also reduce the fear of failing, as there is no such thing as a failed A/B test.
• A/B test is also a risk management tool

How to Conduct an A/B Test

1. Formulate a Hypothesis

A template hypothesis that is helpful for an A/B test

When you successfully state the hypothesis that you want to test, you may come up with an idea of how to test and measure the test. The next sections will hint you for the necessary information.

2. Select the metrics

Some criteria A/b test metrics should acquire

• Easy to understand
• Simple to compute
• Meaningful to stakeholders
• Reliable

Reference metrics for an A/B test

3. Ensure Proper Sampling

Sampling user groups is an important step. The key to this action is FAIR. Only if users are divided fairly into variants is the test validated. Plus, here are some suggestions that I think an A/B test can be based on to sample data:

• Randomizing subsets for users, sessions, or page views is possible.
• Balanced and unbiased groups. For example, if you serve some exclusive user, you must separate them equally to each variant.
• Highly active and low active users should be shared among groups, or the test will not be effectively measured.
• Users all meet the requirements to be qualified for the experiment.
• They were selected at random to prevent sampling errors and avoid drawing conclusions based on a partially represented user population.
• Able to access their track to log their data.
• Balanced or the number of users or samples for each variable is similar.

4. Select a correct A/B Test Type

• Superiority A/B test: This test aims to determine whether the new version performs better than another version or is superior to the control.

• Non-Inferiority A/B test: This test aims to determine whether the new version is not worse than the original version by a margin.

• Balanced A/B test: Prove scenario A is the same as scenario B and vice versa. The metrics change not at all or very little.

• Holdback A/B Test: After releasing a new feature, Keep a subset of users to test without changing the feature after releasing the feature, but this costs much money for a long time

5. Determine When to Stop

• Minimum Detectable Effect (MDE): also referred to as effect size and based on previous tests/target/expectation/insight. This is one of the best tools that I often use for my projects: https://abtestresult.com/sample-size-calculator#parameters-card

• Chance-to-Beat-Original (CTBO): Calculated based on the number of conversion rates and sessions. Its name tells us exactly what we need. Significant results to make the decision are typically <5% (loose) or >95% (win).
• More traffic for an A/B test brings better observation
• If you cannot define a clear duration, no worries. I was in this situation several times (my test had a very low conversion rate = 0.25%—see the above figure) and decided after 3–5 weeks. It is still the ideal time to stop.

Turning Results into Insights

Visualize data Data visualization may bring something from the real data:

A/B test tools

1. VWO (Visual Website Optimizer)

• Advanced audience targeting
• Over 40 platform integrations
• Easy-to-use interface

2. Optimizely

A leader in digital experience platforms, Optimizely supports web and feature experimentation, making it an excellent choice for non-developers. It also offers personalization and content management, providing a well-rounded solution for marketers.

Key Features:

• Supports A/B and multivariate tests
• User-friendly visual editor
• Ideal for non-developers

3. AB Tasty

AB Tasty is an all-in-one conversion optimization platform that combines A/B testing with personalization and audience segmentation. It also features a Bayesian statistics engine for deeper insights into test results.

Key Features:

• Drag-and-drop editor
• Real-time user engagement analytics
• Robust segmentation tools

4. Google Optimize

Google Optimize is a free and accessible tool that integrates seamlessly with Google Analytics. It offers A/B, multivariate, and redirect testing, making it an excellent option for businesses with tighter budgets.

Key Features:

• Free to use
• Visual editor for non-technical users
• Advanced targeting capabilities

* Unfortunately, Google Optimize was stopped by Google in September 2023.

5. Unbounce

Unbounce specializes in creating high-converting landing pages and provides built-in A/B testing tools. Their AI-driven optimization tools are especially useful for marketers looking to streamline their workflows.

Key Features:

• AI-powered tools for optimization
• Real-time analytics
• Tailored for landing page creation
• How to Choose the Right Tool?

Ok, final words — real experiences

• Monitoring A/B test feels like watching a Stock diagram, it is very interesting.
• Sometimes, we may not dig deeply enough into A/B test data and just let the rest of the data be put into the A/B test tool to decide.
• A/B test should be around 3–5 weeks, longer will get polluted samples or deleted cookies, be aware of it.
• If we stop the A/B test after a very short time, we skip the real effect in the future, leading to a wrong decision.
• If running an A/B test longer than 1 month without giving you any information, be brave enough to kill it
• Only make decision superiority test by CTBO >95% or < 5%; sometimes, the trend changes the direction.
• Cultivating a test-friendly culture is a good way to motivate your company.
• If conversion rate is too small (< 1%) we should switch to revenue or other metrics.