A/B Test Significance Calculator
(Two-Proportion Z-Test)
Enter visitors and conversions for your control and challenger to find out whether the difference is statistically significant, with p-value, uplift and a confidence interval. Everything runs in your browser.
Variant A (Control)
Variant B (Challenger)
Confidence Level
A/B Test Result
💡 Formula Guide
What Statistical Significance Means in an A/B Test
An A/B test significance calculator tells you whether the gap between two variants is large enough that random chance is an unlikely explanation. If you split traffic between a control page and a challenger, the two groups will almost never convert at exactly the same rate even when the change does nothing, simply because visitors vary. Significance testing separates a real effect from that background noise.
The result is framed as a p-value: the probability of observing a difference at least this large if both variants actually performed identically. When it falls below your chosen threshold — 0.05 for the common 95% confidence level — the result is called statistically significant. That is a statement about evidence against a no-difference assumption, not a guarantee that the winner will keep winning.
The Math Behind the Calculator
For conversion data the standard method is the two-proportion z-test. With visitors n and conversions x for each variant, the calculator computes each rate p = x ÷ n, then a pooled rate across both groups, and finally a z-score:
The two-tailed p-value is 2 × (1 − Φ(|z|)), with Φ the standard normal cumulative distribution. The critical z values are 1.645 at 90% confidence, 1.960 at 95% and 2.576 at 99%. For a table of converted versus not converted, this test is mathematically equivalent to an uncorrected chi-square test of independence, since χ² = z², so the two names describe the same answer.
Worked Example
Suppose the control page had 5,000 visitors and 200 conversions (4.00%), while the challenger had 5,000 visitors and 250 conversions (5.00%). The relative uplift is (5.00 − 4.00) ÷ 4.00 = 25%. The pooled rate is 450 ÷ 10,000 = 4.5%, the standard error is √(0.045 × 0.955 × (1/5000 + 1/5000)) ≈ 0.004146, and z ≈ 0.01 ÷ 0.004146 ≈ 2.41. That gives a two-tailed p-value of about 0.016 — significant at 95% and at 90%, but not at 99%.
| Confidence | Alpha (threshold) | Critical |z| | Example result (p ≈ 0.016) |
|---|---|---|---|
| 90% | 0.10 | 1.645 | Significant |
| 95% | 0.05 | 1.960 | Significant |
| 99% | 0.01 | 2.576 | Not significant |
The same one-point lift on only 500 visitors per variant would not be significant, which is why sample size matters as much as the observed difference.
Sample Size, Peeking and Other Ways Tests Go Wrong
Most bad A/B decisions come from process, not arithmetic. The first trap is stopping early. If you check a running test every day and stop the moment the p-value dips below 0.05, the real false-positive rate climbs well beyond 5%, because you have given chance many opportunities to look convincing. Decide the sample size up front from your baseline rate and the smallest improvement worth detecting, run the test to that size, and then read the result once.
The second trap is running too short. Behaviour differs across weekdays and weekends, so cover at least one full weekly cycle, and preferably two, even if you hit your visitor target sooner. The third is testing many variants or many metrics. Every extra comparison raises the odds that something looks significant by luck; a correction such as Bonferroni, or a clear single primary metric chosen beforehand, keeps you honest.
Also check the split itself. If you planned 50/50 and one arm received far more traffic, a tracking or bucketing bug may be at fault, and the result cannot be trusted until it is explained. For a rough sense of scale: at a 4% baseline, detecting a lift to 5% with 95% confidence and 80% power takes roughly 6,700 visitors per variant, whereas detecting a lift to 4.4% needs roughly 40,000 per variant. Small effects demand large samples.
How to Use the Calculator
- Enter Variant A. The visitors and conversions for your control, counted over the same period as the challenger.
- Enter Variant B. Visitors and conversions for the challenger. Use unique visitors or sessions consistently for both.
- Pick a confidence level. 95% is the standard default; use 99% for costly or hard-to-reverse changes.
- Read the panel. You get the verdict, both rates, relative uplift, z-score, p-value and a confidence interval for the difference in percentage points.
If you are not sure what your baseline is, start with the Conversion Rate Calculator. To convert a winning variant into money, check its effect on cost per acquisition with the CPA Calculator, and confirm that it does not hurt returns using the ROAS Calculator. For related statistics tools, see the Normal Distribution Calculator, which shows the z and p relationship used here.
Limits of This Calculator
This tool applies a fixed-horizon, frequentist z-test to two independent groups with a binary outcome. It assumes visitors were randomly assigned, each visitor is counted once, and the sample is large enough for the normal approximation — a common guideline is at least about 10 conversions and 10 non-conversions per variant. It does not adjust for multiple comparisons, sequential peeking or Bayesian priors, and it does not handle continuous metrics such as revenue per visitor, which need a different test. Treat the output as a solid first check rather than a substitute for a proper experimentation plan.
Frequently Asked Questions About A/B Test Significance
How do I know if my A/B test result is statistically significant?
Compare the p-value to your threshold. At 95% confidence the threshold is 0.05: a p-value below it means a difference this large would occur less than 5% of the time if both variants truly performed the same. The calculator makes that comparison for you and labels the result Significant or Not Significant.
What does the p-value actually mean?
It is the probability of seeing a gap at least as large as the one observed if there were no real difference between A and B. It is not the probability that B is better, and it is not the chance your result is a fluke. A p-value of 0.03 says the data would be fairly surprising under no-difference; it does not measure how big or valuable the effect is.
Which statistical test does this calculator use?
A two-tailed, two-proportion z-test with a pooled standard error. For a 2×2 table of converted versus not converted, this gives the same result as an uncorrected chi-square test, because chi-square equals z squared. The confidence interval for the difference uses the unpooled standard error.
How many visitors do I need before trusting a test?
It depends on your baseline rate and the smallest lift you care about. Detecting a move from 4% to 5% at 95% confidence and 80% power needs on the order of 6,700 visitors per variant, while a 4% to 4.4% shift needs many times more. Decide the sample size before you start rather than stopping when the numbers look good.
Can I stop the test as soon as it turns significant?
Not safely with a fixed-horizon test. Checking repeatedly and stopping at the first significant reading inflates false positives well above the nominal 5%. Either commit to a sample size in advance, or use a sequential testing method designed for continuous monitoring.
What is the difference between 90%, 95% and 99% confidence?
They set how strict the bar is. Higher confidence means fewer false positives but requires more data to detect the same effect. 95% is the usual default. 90% accepts more false positives, and 99% is stricter, which suits changes that are costly or hard to reverse.
What does the confidence interval tell me that the p-value does not?
It shows the plausible size of the difference. If B beats A and the interval is 0.2 to 1.6 percentage points, the lift is probably positive but could be small. An interval that crosses zero means the data are compatible with no difference or even a decline.
Does statistical significance mean the change is worth shipping?
Not by itself. With very large samples, tiny differences become significant. Check the size of the uplift and the confidence interval against the cost of implementing the change and any effect on revenue per visitor or margin before you roll it out.
Last Updated: October 2026 | Uses the standard two-proportion z-test with a pooled standard error; normal CDF computed with the Abramowitz–Stegun approximation (error under 1.5×10⁻⁷).