A/B test sample size calculator A/B test sample size calculator

The short answer

At a 3% conversion rate, detecting a 10% lift takes 51,487 visitors per variation. On a two-arm test at 1,000 sessions a day, that's 103 days. The number decides whether you can run the test at all, and no tool changes it. Get yours below. Then read on, because the sample size is a floor rather than a schedule, and if your answer is a year, the right move is to test something else.

The calculator

Two inputs to set deliberately. Power is the one most calculators hide, and under-powering is the silent failure: the test runs, shows nothing, the team concludes “no difference.” And check whether the minimum detectable effect is relative or absolute — Evan Miller's calculator defaults to absolute, Optimizely's is labelled relative (both read 26 July 2026), and confusing the two puts you out by an order of magnitude at low baselines. On a 3.0% baseline, +10% relative means 3.30%: a gain of 0.30 percentage points.

The formula, and how it was verified

The literal contents of evanmiller.org/ab-testing/sample-size-fixed.js, fetched 26 July 2026:

function num_subjects(alpha, power_level, p, delta) {
    if (p > 0.5) {
        p = 1.0 - p;
    }
    var t_alpha2 = ppnd(1.0-alpha/2);
    var t_beta   = ppnd(power_level);

    var sd1 = Math.sqrt(2 * p * (1.0 - p));
    var sd2 = Math.sqrt(p * (1.0 - p) + (p + delta) * (1.0 - p - delta));

    return (t_alpha2 * sd1 + t_beta * sd2) * (t_alpha2 * sd1 + t_beta * sd2) / (delta * delta);
}

n per variation = [ z₁₋α/₂ · √(2p(1−p)) + z₁₋β · √( p(1−p) + (p+δ)(1−p−δ) ) ]² ÷ δ²

p is the baseline as a proportion. δ is the absolute effect — compute it from the original baseline (δ = p × relativeMDE) before calling the function, because the p > 0.5 line folds the baseline onto its complement. ppnd is the inverse normal CDF. For one-sided, pass 1 − α instead of 1 − α/2. Round up. His published defaults, from defaults.js in the same fetch: alpha_level = 0.05, power_level = 0.80, one_sided = false, mde_is_relative = 0.

How it was checked. Re-implemented in Node against Evan Miller's own default state (p = 20%, δ = 0.05 absolute, α = 0.05, power = 0.80): 1030.22 raw. His page rounds and displays 1,030; this calculator rounds up to 1,031, because you cannot recruit a fraction of a visitor and rounding down under-powers the test. Expect our numbers to sit one visitor above his. Then reconciled against the standard pooled two-proportion form, which uses the midpoint p̄ = (p₁+p₂)/2 as the null variance rather than the baseline:

Both are defensible. This calculator ships Evan Miller's form; know the spread before you argue with someone else's number.

One more source of confusion: the formula in NIST's e-Handbook §7.2.4.2 is one-sample — a proportion against a fixed target, not two arms against each other — and it drops the two-arm √2 factor. Plenty of blog posts cite it as the A/B formula.

If you test on Optimizely or VWO, use their calculator, not this one. Optimizely's Stats Engine and VWO's enhanced SmartStats are both sequential, and a fixed-horizon number doesn't describe what a sequential engine will do with your data. Both vendors also ship a fixed-horizon mode — Optimizely lists “Frequentist (Fixed Horizon)” alongside Bayesian and Stats Engine, in beta as of 28 July 2026, and VWO's own duration calculator recommends one: “For accurate, error-free results, use the Fixed horizon approach, which only calculates statistics after the full sample size is reached.” If that's what you're running, this number applies. One footnote on Optimizely's own tool: it has no power input at all, and its FAQ carries the heading “Where is statistical power in your sample size calculator?”

How much traffic, and how long

Per variation, α = 0.05 two-sided, 80% power. Computed 26 July 2026 from the formula above.

The columns move faster than the rows because cost is quadratic in the effect. Kohavi, Tang, Xu, Hemkens and Ioannidis, Trials 21:150 (2020): “Because the relationship between the detectable effect and the number of users needed is quadratic, the ability to detect an effect twice as small, e.g., 0.25%, requires quadrupling the number of users.”

As a calendar — two arms, 3% baseline, +10% target, all traffic in the test:

That second column exists because two minimums have nothing to do with sample size. Run whole weeks: shopping has a weekly rhythm, and VWO's duration calculator says so directly — “It is recommended to run your campaigns for at least 7 days to capture natural fluctuations in metric performance across weekdays and weekends” (read 26 July 2026). And run at least two, per Kohavi, Deng, Longbotham and Xu in Seven Rules of Thumb for Web Site Experimenters (KDD 2014), on novelty: “we recommend running experiments for two weeks and looking for such effects.” If your business has a monthly rhythm — payday, a subscription cycle, a Friday drop — run through one.

Longer is not automatically safer. The same authors' 2020 paper puts the final read at “usually based on 1–2 weeks of data.” A fourteen-week requirement is not a fourteen-week test. It's a test you shouldn't run.

totalRequired  = nPerVariation × variations
dailyEntrants  = sessionsPerDay × (allocation ÷ 100)
days           = ceil(totalRequired ÷ dailyEntrants)
runLengthDays  = max(14, ceil(days ÷ 7) × 7)

Why you can't stop when the number looks good

Everything above assumes the sample size was fixed before the test started. Check daily and stop the moment it turns green, and you're running a different experiment from the one the maths describes.

Evan Miller simulated it in How Not To Run An A/B Test (read 26 July 2026): a 50% baseline, no real effect, testing after every observation, stopping at 5% significance or 150 observations. The true false-positive rate came out at 26.1% — “more than five times what you probably thought the significance level was.” To hold a real 5% while looking repeatedly, you have to report at a much tighter threshold:

His fix: “Decide on a sample size in advance and wait until the experiment is over before you start believing the ‘chance of beating original’ figures that the A/B testing software gives you.”

The qualification most pages skip: this is a fixed-horizon frequentist problem. Sequential and Bayesian engines are built precisely so you can look — that is what Optimizely's Stats Engine and VWO's SmartStats are for. Know which you're running on before you take “never peek” as gospel.

The practical fix is to take the decision away from yourself. A Mtrix experiment carries a completion trigger — a visitor count, a date, or a revenue threshold — set when you create it. Put the number from the calculator into the visitor count and stopping stops being a 9pm judgement call. See how experiments are set up.

Mtrix's own reports sit in the second camp: probability-to-be-best is computed with a sequential method built to be monitored while the test runs, the same class as Stats Engine and SmartStats. Watching a Mtrix report mid-test doesn't corrupt it — the trigger is a finish line, not a blindfold.

Revenue per visitor is the expensive metric

Sample-size formulas for conversion rate assume a binary outcome. Revenue per visitor isn't binary and it's violently skewed — one £900 order sits in the same average as four hundred visitors who bought nothing. Skew slows the average's convergence to normality, and Kohavi's Seven Rules puts a number on it (Rule #7, Have Enough Users): the mean needs roughly 355 × s² observations per variant before it is approximately normal, where s is the skewness coefficient. Their Bing table:

Read that third column for what it is: where each metric's average starts behaving normally, not a power calculation. On an equally-sized 50/50 split — what this calculator assumes — the paper says plainly that “Power calculations will typically provide the lower bound for sample size.” What the table shows is the price of skew, because that floor scales with s². Capping is the paper's own mitigation: after capping revenue per user at $10 per user per week, “For the same sample size, Capped Revenue per user can detect a change 30% smaller than Revenue per user.” And, verbatim: “At a commerce site, the skewness for purchases/customer was >10 and for revenue/customer >30.”

For a DTC store: decide the test on conversion rate or add-to-cart rate, and read revenue per visitor as the check that the answer was worth having. Mtrix reports all three on the same experiment, so you don't pick one at setup time.

The skewness figures in that table were corrected to 17.9 and 5.2 in January 2015; the paper's prose still carries the original 18.2 and 5.3.

When the number is bigger than your traffic

This is the real question behind most sample-size searches, and it deserves a straight answer rather than a smaller number.

First, the reframe. Kohavi 2020: “About two-thirds of experiments show that promising ideas that we implemented in products failed to improve the metrics they were designed to change, and this was worse in well-optimized domains such as the search engines, where failures were in the range of 80–90%.” The alternative to testing isn't shipping faster. It's shipping wrong, two times in three. Seven Rules is blunt about the size of real wins, at Bing scale: “For web sites like Bing, where thousands of experiments are being run annually, most fail, and those that succeed improve key metrics by 0.1% to 1.0%, once diluted to overall impact.” Your store isn't Bing, and a first pass at an unoptimised page can beat that. But price a 1% target against the grid above before you plan it.

So change the test, not the statistics.

That last row is where most brands under a thousand orders a month should spend their effort, and capture rate is what makes it work. Mtrix records 100% of sessions — sampling is an opt-in control, not a plan tier — and replay filters by experiment and by variant arm, so when a test does finish you can open the sessions from the losing arm and watch what went wrong. See session replay. Two caveats in writing: replay and error data are kept one month, so a fourteen-week test outlives its own recordings; and Mtrix doesn't run your experimentation programme for you.

The verdict, including the part that costs us the sale. Below roughly a thousand orders a month, most A/B tests you'd want to run cannot conclude inside a quarter. That's not a reason to guess. It's a reason to change what you measure: test changes big enough to matter, judge them on a metric high enough in the funnel to move, and spend the rest of your attention on what actually breaks. If your constraint today is budget rather than capability, Microsoft Clarity is genuinely free, with no traffic ceiling and no card, and it's the honest place to start. Know one limit going in: its playback data is deleted at 30 days (Microsoft's own data-retention doc, read 26 July 2026). Mtrix runs a one-month trial, not a free tier, and it's priced for brands where a percentage point of conversion outweighs the subscription.

Frequently asked questions

Plan the test, then run it without arguing about when to stop

Set the sample size as the finish line when you create the experiment, run every test to it, and read conversion rate, add-to-cart rate and revenue per visitor on the same report.