Skip to content
Really Cool Online Tools logoReally Cool Online Tools

MKT-003

A/B Test Significance Calculator

Version B converted better than version A. The question this answers is whether that would still be true tomorrow, or whether you are looking at the kind of difference that appears by chance when you flip a coin a thousand times twice.

  • Free
  • No signup
  • Private · runs locally

Example

Example figures. Type over them with your own.

Not significant at 95%
No

This difference is still within what chance alone would produce

Confidence
85%

How sure you can be that the difference is real, not that it is large

p-value
0.1496
Rate, version A
5%
Rate, version B
6.5%
Relative change
30%

How much better or worse B is than A, in relative terms

z-score
1.441
How this is calculated
  • Conversion rate = Conversions ÷ Visitors
  • Pooled rate = (Conversions A + Conversions B) ÷ (Visitors A + Visitors B)
  • Standard error = √( pooled × (1 − pooled) × (1 ÷ Visitors A + 1 ÷ Visitors B) )
  • z = (Rate B − Rate A) ÷ Standard error
  • p-value = the two-tailed probability of a z at least that large by chance

How to use it

  1. Enter how many people saw version A, and how many of them converted.
  2. Do the same for version B.
  3. Read the first line: significant at 95%, yes or no.
  4. If it says no, the honest conclusion is that you do not know yet, not that the versions are the same.

What significance does and does not tell you

Significance is about confidence, not about size. A tiny improvement can be significant with enough traffic, and a large one can fail the test with too little. The relative change line is the one that tells you whether the difference is worth having; the significance line tells you whether to believe it.

The 95% threshold is a convention, not a law of nature. A p-value of 0.06 is not meaningfully different from 0.04, and treating one as proof and the other as nothing is the most common way these tests are misread.

The test here is two-tailed, which means it asks whether the versions differ at all rather than whether B beat A specifically. That is the right default, because deciding afterwards which direction you were testing is how a result gets talked into existence.

The mistake that invalidates most tests

Checking the result every day and stopping as soon as it turns significant breaks the maths. Given enough looks, a test between two identical versions will cross the 95% line at some point purely by chance, and stopping there guarantees a false result.

Decide the sample size before starting, then look once. If that is not practical, at least do not stop early on a result you were hoping for.

Frequently asked questions

How much traffic do I need?

It depends on how large the difference is. A change from 5% to 10% shows up in a few hundred visitors per version; a change from 5% to 5.2% can need tens of thousands. The tool warns when the totals are low enough that the answer should not be trusted whatever it says.

It says not significant. Does that mean the versions are the same?

No. It means this data cannot tell them apart. Absence of evidence is not evidence of absence, and the usual next step is more data rather than a conclusion.

What is the p-value?

The probability of seeing a difference at least this large if the two versions were really identical. A p-value of 0.03 means a difference this big would turn up by chance about 3 times in 100. It is not the probability that B is better.

Can I use this for things other than conversions?

Yes, for anything counted as a yes or no out of a total: signups out of visitors, clicks out of impressions, replies out of emails sent. It does not work for averages, such as revenue per visitor, which need a different test.

Other tools people use alongside the A/B Significance.

Browse all marketing tools or see every tool.