A/B Test Significance Calculator

What this tool does

CM8-375 tells you whether an A/B test result is real. It runs a proper two-proportion test, gives the confidence interval on the difference, works out the sample size the test needed, and declines to call a winner where the data cannot support one.

Everything runs inside this single file — no account, no upload, no network request of any kind.

The calculation in full

The pooled standard error is used for the test and the unpooled one for the interval, which is the standard treatment: the test asks whether the two rates could be the same, so it assumes they are; the interval estimates how different they are, so it does not.

What the p-value actually means

It is the probability of seeing a difference at least this large if the two versions were actually identical. A p of 0.03 means a difference this big would turn up three times in a hundred by chance alone.

It is not the probability that the variant is better, and it is not the probability that you are right. At 95% confidence, one test in twenty with no real effect will look like a winner — so a programme running forty flat tests a year should expect two false winners, both of which will be shipped and neither of which will do anything.

Read the interval, not the verdict

The single most useful habit in testing. "Significant" is a yes-or-no answer to a question nobody really has. The interval answers the question you do have: how much better, and how sure are we?

• An interval of +0.1 to +2.4 points is a real effect of unknown size — probably worth shipping, and do not promise anyone the top of the range.
• An interval of −0.3 to +2.9 spans zero. The variant may be better, may be worse, and the test cannot tell.
• An interval of +1.9 to +2.2 is a well-measured effect you can plan around.

A wide interval that happens to sit above zero is a weak result, not a strong one.

Sample size, and why it comes first

Decide the sample size before the test starts, from the smallest lift that would be worth having. The planning table gives it for a range of baseline rates, along with how many weeks it would take at your traffic.

The arithmetic is unforgiving in two ways. Halving the effect you want to detect quadruples the sample, and lower baseline rates need far more traffic — detecting a tenth improvement on a 1% rate needs roughly ten times the visitors of the same improvement on a 10% rate.

The honest consequence is that most sites cannot test most things. If the planning table says forty weeks, the answer is not to run it for four and squint at the result — it is to test something with a bigger expected effect, or to make the decision on judgement and say so.

Peeking, and why it breaks everything

Every A/B test wanders. Watch a test with no real effect and it will drift across the significance line several times before settling. Stop it the first time it crosses and you will find a "winner" in something like a third of tests where there is nothing at all.

This is not a small correction. It is the difference between a testing programme that improves things and one that generates a stream of confident, worthless changes.

The arithmetic cannot detect it. A test stopped early produces a perfectly valid looking p-value. That is why the tool asks you to declare it, and why a declared early stop overrides the verdict entirely. The sample includes one: a pricing test called after four days at a sixth of its needed sample, which is the classic false winner.

If you must look early, use a sequential testing method designed for it, or accept a much stricter threshold. Do not use this calculator and stop when it goes green.

Too few conversions

Below five conversions in either variant, the normal approximation behind all of this stops being reliable, and the tool reports the figures without a verdict.

Three conversions against seven looks like a doubling and is entirely consistent with no effect at all. This is where more testing programmes go wrong than anywhere else, because the numbers are small enough to feel readable.

Relative against absolute

The lift column is relative: 3.0% to 3.3% is a 10% lift. The interval is in percentage points: the same change is +0.3 points.

Both are correct and they get confused constantly, usually in the direction that makes a result sound bigger. A "40% improvement" on a 0.5% conversion rate is 0.2 points, which may be worth almost nothing. The annual value column exists to settle that argument in money.

Write the hypothesis first

The tool will not save a test without one, and that is deliberate. A hypothesis written afterwards is a story, and a test without one is a search for something interesting — where something interesting is always found.

A good hypothesis names the audience, the mechanism and the expected direction: "visitors from paid search do not know the category, so a benefit-led headline should convert better". It also makes a failed test informative, because you learn the mechanism was wrong.

Record the tests that lost

Keep the flat results and the ones where the original won. Three reasons: they stop the same idea being retested every year; the ratio of winners to tests is the honest measure of a testing programme; and a variant that lost significantly is a genuine finding. The sample has one — a longer free trial reduced signups, which is worth more than most wins.

A programme reporting mostly winners is not a good programme. It is one calling results too early. Between one in five and one in three is what honest testing looks like.

Mistakes the arithmetic cannot see

• An uneven split. If one side has three times the traffic, something is usually broken. The tool warns, and cannot tell you which side is wrong.
• Not running whole weeks. Weekday and weekend traffic behave differently; a test covering nine days weights one weekend twice.
• Visitors counted more than once, or seeing both versions on different devices.
• Something else changing mid-test — a campaign starting, a price changing, a holiday.
• Testing many things at once and reporting whichever won. Twenty metrics at 95% confidence produce one false winner by definition.

What this does not do

• It compares two variants. For three or more, the false-positive rate rises with every comparison and needs a correction this tool does not apply.
• It handles rates, not values. A variant with fewer but larger orders needs a different test.
• It says nothing about what happens next. A signup test that wins can lose on quality, as the sample form test did.
• It cannot detect a broken test, and a broken test produces a confident answer.

Printing and sharing

The Report tab prints the tiles, charts and both tables with a title block you fill in. The results table shows every statistic, so a result can be checked rather than believed; the planning table is what to consult before agreeing to run anything.

Saving your work

Tests are held in this browser, on this computer, and stay there between visits. Use the backup button to write a JSON file you control — a test archive is worth keeping for years.

Accuracy & disclaimer

The formulas and critical values are stated in full above so any figure can be checked. What the tool cannot check is whether the test was run properly, and a badly run test produces output that looks exactly like a good one.