MKT-003
A/B Test Significance Calculator
Version B converted better than version A. The question this answers is whether that would still be true tomorrow, or whether you are looking at the kind of difference that appears by chance when you flip a coin a thousand times twice.
- Free
- No signup
- Private · runs locally
Example
Example figures. Type over them with your own.
- Not significant at 95%
- No
- Confidence
- 85%
- p-value
- 0.1496
- Rate, version A
- 5%
- Rate, version B
- 6.5%
- Relative change
- 30%
- z-score
- 1.441
This difference is still within what chance alone would produce
How sure you can be that the difference is real, not that it is large
How much better or worse B is than A, in relative terms
How this is calculated
- Conversion rate = Conversions ÷ Visitors
- Pooled rate = (Conversions A + Conversions B) ÷ (Visitors A + Visitors B)
- Standard error = √( pooled × (1 − pooled) × (1 ÷ Visitors A + 1 ÷ Visitors B) )
- z = (Rate B − Rate A) ÷ Standard error
- p-value = the two-tailed probability of a z at least that large by chance
How to use it
- Enter how many people saw version A, and how many of them converted.
- Do the same for version B.
- Read the first line: significant at 95%, yes or no.
- If it says no, the honest conclusion is that you do not know yet, not that the versions are the same.
What significance does and does not tell you
Significance is about confidence, not about size. A tiny improvement can be significant with enough traffic, and a large one can fail the test with too little. The relative change line is the one that tells you whether the difference is worth having; the significance line tells you whether to believe it.
The 95% threshold is a convention, not a law of nature. A p-value of 0.06 is not meaningfully different from 0.04, and treating one as proof and the other as nothing is the most common way these tests are misread.
The test here is two-tailed, which means it asks whether the versions differ at all rather than whether B beat A specifically. That is the right default, because deciding afterwards which direction you were testing is how a result gets talked into existence.
The mistake that invalidates most tests
Checking the result every day and stopping as soon as it turns significant breaks the maths. Given enough looks, a test between two identical versions will cross the 95% line at some point purely by chance, and stopping there guarantees a false result.
Decide the sample size before starting, then look once. If that is not practical, at least do not stop early on a result you were hoping for.
Frequently asked questions
How much traffic do I need?
It depends on how large the difference is. A change from 5% to 10% shows up in a few hundred visitors per version; a change from 5% to 5.2% can need tens of thousands. The tool warns when the totals are low enough that the answer should not be trusted whatever it says.
It says not significant. Does that mean the versions are the same?
No. It means this data cannot tell them apart. Absence of evidence is not evidence of absence, and the usual next step is more data rather than a conclusion.
What is the p-value?
The probability of seeing a difference at least this large if the two versions were really identical. A p-value of 0.03 means a difference this big would turn up by chance about 3 times in 100. It is not the probability that B is better.
Can I use this for things other than conversions?
Yes, for anything counted as a yes or no out of a total: signups out of visitors, clicks out of impressions, replies out of emails sent. It does not work for averages, such as revenue per visitor, which need a different test.
Related tools
Other tools people use alongside the A/B Significance.
- MKT-001UTM BuilderBuild correctly encoded UTM tracking links for your campaigns.Open
- CAL-002Profit Margin CalculatorCalculate profit, profit margin and markup from your cost and selling price.Open
- CAL-009Compound Interest CalculatorSee what savings grow to, with regular contributions and any compounding frequency.Open
Browse all marketing tools or see every tool.