A/B Test Significance Calculator

Free tool by AAJ — find out whether you have a winner or just noise

Enter visitors and conversions for your control and up to three variants and get a verdict: a two-proportion z-test with p-values and confidence intervals, a Bayesian probability that each version is the true best, and a Bonferroni correction applied automatically the moment you test more than one variant. A second tab sizes the test before you run it. The calculation happens in your browser.

Use the tool at aajconsult.com/tools/ab-test-significance-calculator.

What "significant" actually means here

The calculator runs a two-proportion z-test between the control and each variant and reports a p-value against your chosen confidence level — 90%, 95% or 99%, with 95% as the default. It defaults to two-tailed, which is the honest choice: a one-tailed test assumes in advance that your variant cannot be worse, and variants are worse all the time.

Alongside the frequentist verdict it reports P(best) — the Bayesian probability that each version is the genuine top performer. The two answer different questions. The p-value asks how surprising your result would be if the versions were identical. P(best) asks, given the data, which version you should bet on. A test can be inconclusive on the first and still give you a 92% read on the second, which is often the number a decision actually needs.

Why testing three variants changes the maths

Every comparison you add is another chance to find a winner that is not there. Test three variants at 95% confidence and the probability of at least one false positive is far higher than 5% — which is how teams end up shipping a "winner" that quietly reverses in production.

So the calculator tightens the threshold automatically. With more than one variant it divides your alpha by the number of variants — a Bonferroni correction — and shows you the adjusted threshold it is now testing against, so your overall false-positive rate stays where you set it. You do not have to remember to do this, and you cannot forget.

The warnings are part of the tool

If a variant has fewer than 1,000 visitors, it says so: results at that volume are noisy regardless of what the p-value reads. If conversions exceed visitors, it flags the data rather than computing something confident from an impossible input. A calculator that returns a clean verdict on broken numbers is worse than no calculator.

Sizing the test before you run it

The second tab works the other way round. Give it your baseline conversion rate, the smallest relative lift worth detecting, how many variants you are running, your power — 80% or 90% — and your confidence level, and it returns the visitors you need per variant, the total across the test, and how long that takes at your daily traffic. Durations of two weeks or more are shown in weeks, because that is how you will actually plan them.

This is the tab that prevents most wasted tests. A 3% baseline and a 10% relative lift needs a great deal more traffic than teams expect, and finding that out beforehand is the difference between a test that concludes and a test that gets called early on a hunch. Whatever the number says, run for at least one full business cycle — usually one to two weeks — even if you hit the sample size sooner, because a Tuesday-to-Thursday audience is not your whole audience.

What you get back

A verdict headline that names the winner, names a significant loser, or says the test is inconclusive rather than dressing up a near-miss. A results table with conversion rate, uplift, p-value, P(best) and a plain yes or no per variant. A chart of each version's conversion-rate distribution, and confidence intervals at your chosen level — the interval usually teaches more than the p-value, because a "winner" whose interval spans zero to fourteen percent is not a result you should plan a roadmap around.

There is a copy button for the summary and a share link that carries your inputs, so the result can go into a document or a thread without a screenshot.

Frequently asked questions

How many variants can I test?

A control plus up to three variants. Past that, the correction gets punishing and the traffic requirement usually makes the test impractical — you are better off running fewer, braver variants.

What confidence level should I use?

95% unless you have a reason. 90% finds more winners and more false ones; 99% is for changes that are expensive to reverse.

Should I use one-tailed to reach significance faster?

No. It reaches significance faster by assuming your variant cannot lose, which is the assumption a test exists to check.

Is my data stored?

The calculation runs entirely in your browser. There is an optional email unlock on the results panel, and if you use it your test figures are saved with your email — there is a skip link next to it, and skipping keeps everything on your machine. The sample-size tab is never gated.

Related resources

Marketing Maturity Scorecard — Unit Economics Calculator — All free tools — Analytics, Experiments & Budget hub

Significance is the easy half.

Knowing which test to run next — and which result is worth a roadmap — is the part a calculator cannot do. AAJ builds the experiment programme around it. Book a Growth Audit.

Run this in your agent

The same test runs as the free AAJ agent skill A/B Test Significance, so a coding agent can check a result against your own data. Install it with npx skills add sarojkjha/aaj-marketing-skills --skill ab-test-significance.