A/B testing shows two versions of a page, email or advert to similar audiences at the same time and compares which one produces more of a chosen action. To do it properly, change one thing, state a hypothesis, split traffic randomly, run the test long enough to gather meaningful data and decide using your pre-set goal rather than gut feeling.
- Test one meaningful change at a time against a single success measure.
- Write a hypothesis before the test so the result teaches you something.
- Run versions simultaneously and let the test run for a full business cycle.
- Low-traffic sites should prioritise obvious fixes over formal tests.
- Record every test, including the ones that did not win.
What is A/B testing?
In an A/B test, version A is your current page, email or advert, and version B is a variation with one deliberate difference. Visitors are split randomly between the two, and you measure which version leads to more of the action you care about, such as a form submission, purchase or click. Because both run at the same time, outside factors affect them equally.
This is more reliable than changing a page and comparing this month with last month, because seasons, festivals, ad budgets and news can all shift behaviour. A/B testing isolates the effect of your change. It turns arguments about opinions, such as which headline is better, into a question that real visitor behaviour can answer.
What should you test first?
Begin with elements that influence the main decision: the headline, the main call to action, the offer, the form length, the product page layout or the subject line of an important email. Small cosmetic tweaks such as a slightly different shade of blue rarely matter as much as clearer messaging or fewer steps in a form.
Prioritise ideas using a simple scoring method. Ask how much impact the change could have, how confident you are that it addresses a real problem and how easy it is to build. Evidence from analytics, customer questions and session recordings should inspire your ideas. A test that comes from an observed problem is more likely to teach you something useful.
- Headline and opening paragraph of a landing page.
- Primary button text and placement.
- Offer framing, such as free consultation versus a free quote.
- Number of fields in an enquiry form.
- Email subject lines and sending times.
How do you write a good hypothesis?
A hypothesis has three parts: what you will change, what you expect to happen and why. For example: because visitors ask about delivery time before ordering, showing estimated delivery on the product page should increase add-to-cart actions. This format makes you commit to a reason, so that a win or loss improves your understanding of customers.
Decide before the test begins what your primary measure is and what you will do with each outcome. Choosing a metric after seeing the results invites self-deception, since almost any experiment can be made to look successful by picking the right number. Keep secondary measures, such as bounce or revenue per visitor, as guardrails to confirm that a win is not causing harm elsewhere.
How long should a test run, and how much traffic is enough?
Run a test long enough to include a normal cycle of your business. Many businesses see different behaviour on weekdays and weekends, or at the start and end of the month, so cutting a test short after a couple of lucky days is risky. Online calculators can estimate the sample you need based on your current conversion rate and the size of improvement you hope to detect.
If your site has little traffic, a formal test could take months and still be inconclusive. In that case concentrate on clear fixes, such as removing friction, speeding up pages and answering common objections, and judge them with before-and-after observation. Save controlled tests for pages with enough visits to reach meaningful conclusions in a reasonable time.
What mistakes ruin A/B tests?
The most frequent mistake is stopping a test as soon as one version looks ahead. Early results swing wildly and often reverse. Another is changing several things at once, which makes it impossible to know what caused the difference. Others include testing during an unusual event, such as a major sale, and then assuming the result applies all year, or running overlapping tests that influence the same visitors.
Technical problems also hide in plain sight. Flickering pages, tracking that fails on one version, or traffic that is not split evenly can invalidate everything. Run a quick check before launching: confirm that both versions display properly on mobile and desktop, that goals are recorded and that visitors see a consistent version on return visits.
- Stopping the test early because one version seems to win.
- Changing multiple elements in the same variation.
- Choosing the success metric after the test.
- Ignoring mobile display and page speed differences.
- Running overlapping tests on the same audience.
What do you do after the test ends?
If the variation wins, implement it for everyone and monitor results to confirm they hold. If it loses or shows no clear difference, that is still valuable: you have learned that this change did not matter, which frees you to focus elsewhere. Write down the hypothesis, the setup, the result and what you will try next.
Over time this record becomes a knowledge base about your customers. You will see patterns, such as clear pricing information outperforming clever slogans, that guide design decisions across the whole business. Share findings with sales, support and marketing teams so that insights from testing improve every customer touchpoint, not just one page.
Step by step
- Pick one goal. Choose a single primary measure, such as enquiries submitted or orders completed.
- Form a hypothesis. Write what you will change, what you expect and the reason, based on real evidence.
- Build the variation. Create version B with one meaningful difference and check it on all devices.
- Split traffic and run. Send visitors randomly to each version at the same time and let the test run for a full business cycle.
- Analyse and record. Compare results against your goal, implement or discard the change and log what you learned.
Frequently asked questions
Can I run A/B tests on a small website?
You can, but results may take a long time to become clear. With limited traffic, concentrate on obvious usability and message fixes and test only your most important page.
What is the difference between A/B and multivariate testing?
A/B testing compares two whole versions with a single difference. Multivariate testing changes several elements in combination and needs much more traffic to interpret.
Do A/B tests work for emails and ads?
Yes. Subject lines, preview text, creative and calls to action are all commonly tested. Split a portion of your audience, pick the winner by a pre-set goal and send it to the rest.
Is it bad if a test shows no difference?
No. A flat result tells you that the element is not a lever for your audience, and it saves you from investing further in that idea.
Need help with this? See our Conversion Rate Optimisation service or talk to Yash Parikh.