A/B test significance calculator — and the checks behind the verdict A/B test significance calculator — and the checks behind the verdict

The short answer

You ran the test. Four numbers — visitors and conversions for each arm — tell you whether the difference is bigger than chance would produce: rates, z-score, p-value, confidence interval, verdict. Two integrity checks ride along that most calculators skip: a sample-ratio-mismatch test on your split and a low-data guard (ABTestGuide is the exception, credited below). What no calculator can tell you: a p-value is not the probability you're right, and significant is not the same as worth shipping.

The calculator

The defaults are the genre's: two-sided, 95% confidence — the same defaults Evan Miller's defaults.js publishes and ABTestGuide offers (both read 26 July 2026). The intended-split control feeds the mismatch check; leave it at 50/50 unless you allocated differently.

How it's calculated

ABTestGuide publishes its maths on the page — rare in this genre — so this calculator implements the same unpooled two-proportion z-test, quoted from abtestguide.com/calc (read 26 July 2026):

“Standard error A = ( CR A * (1-CR A ) / Visitors A )^1/2”

“Std. Error of difference = ( SE A² + SE B² )^1/2”

“Z-score = ( CR B - CR A ) / SE difference”

“Relative uplift in Conversion Rate = CR B - CR A / CR A”

The p-value reads that z off the normal curve — two-sided p = 2·(1 − Φ(|z|)) — and the interval on the difference is (CR B − CR A) ± z·SE of difference. Units, once and everywhere: the interval is absolute, in percentage points; the uplift is relative and labelled relative.

Worked example, computed in Node, 28 July 2026 — also the calculator's default state:

Relative uplift +15.0%. Standard error of the difference 0.0025. z = 1.80, two-sided p = 0.0717 — not significant at 95%. The 95% interval runs −0.04 to +0.94 percentage points: it crosses zero, so “B is 15% better” is on the table and so is “B does nothing.” The same data under other settings:

A verdict that flips with the settings is a p-value near the line; the honest reading is: keep collecting. One-sided gets there faster by giving up detecting that B made things worse — that trade, and everything else you decide before a test starts, lives on the sample size calculator.

What significance doesn't mean

Statistical significance says the difference probably isn't luck — nothing about whether it matters. Give a 3.00%-versus-3.02% test 5.6 million visitors an arm and it turns green (computed with the same script); nothing about the green makes 0.02 points worth a redesign. And p = 0.0018 is not a 99.8% chance that B is better — it's how often data this extreme appears when the variants convert identically.

So decide on the interval. With 380 conversions for B instead of 345: z = 3.12, p = 0.0018, 95% interval +0.30 to +1.30 points. The worst plausible case is a tenth of your conversion rate — shippable not because it's significant, but because its floor is worth having.

When you can trust a green result

Three checks, in order.

1. The planned sample was reached. A fixed-horizon test assumes the sample size was set in advance; check repeatedly and stop on the first green and the nominal 5% false-positive rate becomes a measured 26.1% (Evan Miller's simulation in How Not To Run an A/B Test, read 26 July 2026). Plan the number on the sample size calculator — that page owns the argument — and run to it.

2. The split matches what you intended. Kohavi, Deng, Longbotham and Xu list “the sample ratios matched the expected ratios” among their own preconditions (Seven Rules of Thumb for Web Site Experimenters, KDD 2014). A 50/50 test that produced 10,000 and 11,000 visitors did not have bad luck — the check here returns p ≈ 5×10⁻¹² on that split. It has a bug: a redirect, a bot filter or a broken variant dropped sessions from one arm, and rarely at random. ABTestGuide's alert says a mismatch “indicates there might be a problem with your distribution” (read 26 July 2026); this one tests your split with a chi-square and alerts below p = 0.01. A failed check doesn't shade the verdict. It voids it.

3. Enough conversions. ABTestGuide again, verbatim: “This is a significant result based on your thresholds, but the actual weighted difference is 20 conversions or less - please be careful interpreting these results - gathering more data is advised” (read 26 July 2026). This calculator carries the same guard at the same threshold.

The report that never sends you to a calculator

This page is fixed-horizon frequentist — the z-test the keyword asks for, with the rule attached: fix the sample, wait, then look. Mtrix experiment reports don't make you live that way: probability-to-be-best is computed with a sequential method designed to be monitored while the test runs — the same class of statistics as Optimizely's Stats Engine and VWO's SmartStats. Every experiment carries a completion trigger — visitor count, date or revenue — so stopping is a setup decision, not a judgement call. When a test concludes, session replay filters by experiment and by variant arm: open the sessions from the losing variant and watch why it lost. Scoped honestly: if you test on Optimizely or VWO, use their calculators — their engines are sequential, and a fixed-horizon number doesn't describe what they'll do with your data.

Frequently asked questions

Read the verdict, then watch the sessions behind it

Set the finish line when you create the experiment, let the report carry the statistics, and open the sessions behind whichever arm loses.