A/B Test Significance Calculator
Compare two versions with a two-proportion z-test to see if the difference is significant at 95 per cent confidence.
- Conversion rate A
- —
- Conversion rate B
- —
- Uplift of B over A
- —
- Verdict
- —
Planning approximation using a normal approximation; very small samples or very low conversion counts make the z-test less reliable.
What is the A/B Test Significance Calculator?
This calculator compares the conversion rates of version A and version B using a two-proportion z-test. If the absolute z-score is 1.96 or more, the difference is called significant at 95 per cent confidence, meaning chance alone is an unlikely explanation. Significance does not prove the change will pay off.
How it works
Rate = conversions / visitors for each version. Pooled rate p = (conversions A + conversions B) / (visitors A + visitors B). Standard error = square root of p x (1 - p) x (1/visitors A + 1/visitors B). z = (rate B - rate A) / standard error. Significant when |z| is at least 1.96.
How to use it
- Enter visitors and conversions for version A.
- Enter visitors and conversions for version B.
- Read the rates, uplift and z-score.
- Check the verdict, and avoid stopping the test early just because it looks significant.
A/B Test Significance Calculator — FAQ
Does significant mean B is better?
It means the gap is unlikely to be random at this confidence level. Check the direction of the uplift and whether the size matters to your business.
Can I peek at results daily?
Repeatedly checking and stopping at the first significant result raises the chance of false wins. Decide the sample size beforehand and wait for it.
What if the result is not significant?
It means there is not enough evidence of a difference. You may need more visitors, a bigger change or a longer test.