A/B testing is a controlled experiment where two versions of a web page, email or ad are shown to randomly assigned groups, and you measure which performs better. Version A is the control, version B has one change. Because the groups are random, the difference in results can be attributed to the change.
It is the core of conversion rate optimisation. It replaces opinions with data and stops you rolling out changes that make things worse.
What can you test?
- Web pages: headlines, images, CTAs, forms, pricing, content order.
- Ecommerce: product pages, cart, shipping information, checkout.
- Email: subject line, sender name, send time, content.
- Ads: copy, images and landing pages, using platform experiments.
How to run an A/B test
- Start with data from analytics, heatmaps and session recordings.
- Write a hypothesis with a reason.
- Choose one primary metric and a few secondary ones.
- Calculate sample size before you start.
- Run for at least one to two full weeks and until the sample is reached.
- Analyse by device and traffic source, and document the learning.
Statistical significance
Significance tells you how likely it is that the difference is due to chance. The usual threshold is 95 %.
| Baseline conversion rate | Lift to detect | Approx. sample per variant |
|---|---|---|
| 2 % | 20 % (to 2.4 %) | about 21,000 visits |
| 5 % | 20 % (to 6 %) | about 8,000 visits |
| 10 % | 10 % (to 11 %) | about 15,000 visits |
Rounded estimates at 95 % significance and 80 % power. Use a calculator for your own case.
Common mistakes
- Stopping when the result first looks significant.
- Changing many things in one variant.
- Too little traffic for small changes.
- Ignoring flicker.
- Ignoring order value.
Tools
Google Optimize was shut down on 30 September 2023. Alternatives include VWO, AB Tasty, Optimizely, Adobe Target, Convert and PostHog, plus Google Ads experiments and Meta A/B tests for ads. Testing tools set cookies and require consent.
Frequently asked questions
What is A/B testing?
An experiment where two versions are shown to randomly assigned groups to see which performs better.
How long should an A/B test run?
At least one to two full weeks and until the planned sample size is reached.
What replaced Google Optimize?
Optimize was shut down in September 2023. Alternatives include VWO, AB Tasty, Optimizely, Adobe Target, Convert and PostHog.
Read also
Sources
- Google, Sunset of Google Optimize: support.google.com/optimize/answer/12979939
- Evan Miller, Sample Size Calculator: www.evanmiller.org/ab-testing/sample-size.html
