A/B Test Significance Calculator
Tests whether the gap between two variants is bigger than chance would explain, and shows how many visitors a test needs.
Share sends a link that reopens these numbers. Copy details pastes the full breakdown as a list.
The formula
pooled = (cA + cB) ÷ (nA + nB) SE = √(pooled × (1 − pooled) × (1 ÷ nA + 1 ÷ nB)) z = (cB ÷ nB − cA ÷ nA) ÷ SE p = 2 × (1 − Φ(|z|)) sample size per variant = (zα × √(2p̄q̄) + zβ × √(p₁q₁ + p₂q₂))² ÷ (p₂ − p₁)²
This is a two-proportion z-test. The pooled rate assumes no real difference, and p is how often a gap at least this large would appear by chance if that were true. In the sample size formula, p₁ is A’s rate, p₂ is that rate with the uplift added, p̄ is their mean and q = 1 − p.
What to watch out for
Work out the sample size before you start and run the test until you reach it. Peeking at a test early and stopping as soon as it looks significant inflates false positives, because a run of random results will cross the line sooner or later.
Statistical significance is not the same as a worthwhile difference. A very large test can prove a tiny uplift is real when it is too small to pay for the change. Visitors are assumed to be split at random and counted once.
Frequently Asked Questions
What does the p-value mean?
It is how often a gap at least this large would appear by chance if the variants were really the same. A z-score of 1.96 gives p = 0.05, the usual line for the 95% level. It is not the chance that B is better, even though 1 − p is often called confidence.
Is my A/B test significant?
Compare the p-value with your level. 480 of 12,000 against 570 of 12,000 gives z = 2.84 and p = 0.0045, significant at 95%, with an uplift of 18.8%. 40 of 1,000 against 44 of 1,000 is an uplift of 10.0%, but p = 0.66, so it is not.
How many visitors do I need per variant?
It depends on the base rate and the smallest uplift worth finding. To detect a 20% uplift on a 5% rate at 95% confidence and 80% power takes about 8,158 visitors per variant. Halving the uplift roughly quadruples that.
Does significant mean worth doing?
No. 40,000 against 40,600 conversions from 1,000,000 visitors each is significant at p = 0.031, yet the uplift is only 1.5%. Check that it pays for the change before you roll it out.