A/B Test Calculator
This A/B test calculator tells you whether the difference between two conversion rates is statistically significant or just noise. Enter the number of visitors and the number of conversions for your control (variant A) and your challenger (variant B), and the calculator works out each conversion rate, the relative change between them, then runs a two-proportion z-test to produce a z-score and a two-tailed p-value. If the p-value comes in under 0.05 you can call the result significant at the 95 percent confidence level, which is the standard bar most marketers and product teams use before rolling a winner out. Anyone running experiments uses this: an online shop testing a new checkout button, a SaaS team trialling different onboarding copy, or an email marketer comparing two subject lines. The maths is the same whatever the channel, as long as each visitor sees only one variant and conversions are counted per visitor, not per page view. The calculator also shows how confident you can be that the difference is real, which helps when a test lands just short of the bar. Keep tests running until both variants have a decent sample, because small samples make even large differences hard to trust.
Variant B converts at 5.60% versus 5.00% for A, a relative change of +12.0%. With z = 1.89 the two-tailed p-value is 0.058, which is not significant at the 95% confidence level (about 94.2% confidence that the difference is real).
Uses a pooled two-proportion z-test with a two-tailed p-value. Each visitor should see only one variant, and each visitor should be counted once. Peeking at results early and stopping the moment significance appears inflates false positives.
How it works
Each conversion rate is simply conversions divided by visitors. To test whether the two rates genuinely differ, the calculator pools all conversions over all visitors into a single pooled rate, then computes the standard error of the difference: the square root of the pooled rate times one minus the pooled rate, times the sum of one over each sample size. The z-score is the difference between the two rates divided by that standard error, and the two-tailed p-value is the probability of seeing a z-score at least that far from zero if the variants really performed identically. A p-value below 0.05 is the conventional cut-off for significance at the 95 percent confidence level. The test assumes independent visitors randomly assigned to one variant each, and it is a two-tailed test, so it detects a difference in either direction.
Worked example
Variant A: 10,000 visitors and 500 conversions, a rate of 5.00%. Variant B: 10,000 visitors and 560 conversions, a rate of 5.60%, a relative change of +12.0%. The pooled rate is 1,060 / 20,000 = 5.30%. The standard error is the square root of 0.053 x 0.947 x (1/10,000 + 1/10,000), which is 0.00317. z = (0.056 - 0.050) / 0.00317 = 1.89, and the two-tailed p-value is 0.058. That misses the 0.05 cut-off, so the uplift is not yet significant at the 95 percent level: you can be about 94.2% confident the difference is real, and the sensible move is to let the test keep running rather than call a winner.
Related calculators
- P-Value Calculator: convert test statistics to p-values.
- Sample Size Calculator: how many visitors you need.
- Z-Test Proportion Calculator: one-sample proportion test.
- Chi-Square Test of Independence Calculator: the equivalent test in table form.
- Margin of Error Calculator: uncertainty around a single rate.