A/B Test Significance Calculator

Free tool by AAJ — find out whether you have a winner or just noise

Enter visitors and conversions for your control and up to three variants and get a verdict: a two-proportion z-test with p-values and an interval on the difference between each variant and control (the same test as AAJ's ab-test-significance agent skill), a Bayesian probability that each version is the true best, and a Bonferroni correction applied automatically the moment you test more than one variant. A second tab sizes the test before you run it. The calculation happens in your browser, and the results are shown without asking for an email.

Use the tool at aajconsult.com/tools/ab-test-significance-calculator.

What "significant" actually means here

The calculator runs a two-proportion z-test between the control and each variant and reports a p-value against your chosen confidence level — 90%, 95% or 99%, with 95% as the default. It defaults to two-tailed, which is the honest choice: a one-tailed test assumes in advance that your variant cannot be worse, and variants are worse all the time.

Alongside the frequentist verdict it reports P(best) — the Bayesian probability that each version is the genuine top performer. The two answer different questions. The p-value asks how surprising your result would be if the versions were identical. P(best) asks, given the data, which version you should bet on. A test can be inconclusive on the first and still give you a 92% read on the second, which is often the number a decision actually needs.

Why testing three variants changes the maths

Every comparison you add is another chance to find a winner that is not there. Test three variants at 95% confidence and the probability of at least one false positive is far higher than 5% — which is how teams end up shipping a "winner" that quietly reverses in production.

So the calculator tightens the threshold automatically. With more than one variant it divides your alpha by the number of variants — a Bonferroni correction — and shows you the adjusted threshold it is now testing against, so your overall false-positive rate stays where you set it. You do not have to remember to do this, and you cannot forget.

The warnings are part of the tool

If a variant has fewer than 1,000 visitors, it says so: results at that volume are noisy regardless of what the p-value reads. If conversions exceed visitors, it flags the data rather than computing something confident from an impossible input. A calculator that returns a clean verdict on broken numbers is worse than no calculator.

Sizing the test before you run it

The second tab works the other way round. Give it your baseline conversion rate, the smallest relative lift worth detecting, how many variants you are running, your power — 80% or 90% — and your confidence level, and it returns the visitors you need per variant, the total across the test, and how long that takes at your daily traffic. Durations of two weeks or more are shown in weeks, because that is how you will actually plan them.

Size the test before you run it. A 3% baseline and a 10% relative lift needs about 53,000 visitors per variant at 95% confidence and 80% power, and finding that out beforehand is the difference between a test that concludes and a test that gets called early on a hunch. Whatever the number says, AAJ's view is to run for at least one full week even if you hit the sample size sooner, because a Tuesday-to-Thursday audience is not your whole audience.

What you get back

A verdict headline that names the winner, names a significant loser, or says the test is inconclusive rather than dressing up a near-miss. A results table with conversion rate, uplift, the difference from control with an interval that agrees with the test, p-value, P(best) and a plain yes or no per variant. A chart of each version's conversion-rate distribution, and confidence intervals at your chosen level — the interval usually teaches more than the p-value, because a "winner" whose interval spans zero to fourteen percent is not a result you should plan a roadmap around.

There is a copy button for the summary and a share link that carries your inputs, so the result can go into a document or a thread without a screenshot.

Frequently asked questions

What does statistical significance mean in an A/B test?

Statistical significance means the difference in conversion rates between your variant and control is unlikely to be due to random chance. At 95% confidence, a significant result means there is less than a 5% probability you would see a difference this large if the two versions actually performed the same.

How do I calculate A/B test significance?

Use a two-proportion z-test. Compute the pooled conversion rate across both groups, the standard error, then the z-score (difference in rates divided by standard error). Convert z to a p-value using the normal distribution. If p is less than your significance threshold (commonly 0.05), the result is statistically significant. This calculator does that math automatically.

What's the difference between frequentist and Bayesian A/B testing?

Frequentist testing answers "if there were truly no difference, how likely is a result this extreme?" via p-values. Bayesian testing answers "given the data, what's the probability each variant is the true best?" using posterior distributions. They answer different questions — when both agree, you can act with confidence. This calculator reports both.

How many visitors do I need for an A/B test?

It depends on your baseline conversion rate and the minimum detectable effect you care about. As a rule of thumb, detecting a 10% relative lift on a 3% baseline at 95% confidence and 80% power needs roughly 50,000 visitors per variant. Use the sample-size planner in this calculator to size your specific test.

What is the Bonferroni correction and when do I need it?

When you test multiple variants against one control, every additional comparison inflates the chance of a false positive. The Bonferroni correction divides your significance threshold by the number of comparisons (so for 3 variants at 95% confidence, the per-test threshold becomes p < 0.0167 instead of 0.05). This calculator applies it automatically when you add more than one variant.

Can I stop an A/B test as soon as it reaches significance?

No. Stopping the moment a test crosses significance — called peeking — dramatically inflates the false-positive rate. Set your sample size up front using a sample-size calculator, run the test for at least one full business cycle (1–2 weeks), and only then call the result.

Related resources

Marketing Maturity Scorecard — Unit Economics Calculator — All free tools — Analytics, Experiments & Budget hub

Significance is the easy half.

Knowing which test to run next — and which result is worth a roadmap — is the part a calculator cannot do. AAJ builds the experiment programme around it. Book a Growth Audit.

Run this in your agent

The free AAJ agent skill A/B Test Significance does this job inside a coding agent, on your own data. Install it with npx skills add sarojkjha/aaj-marketing-skills --skill ab-test-significance.

Using Claude or ChatGPT instead? Add the AAJ engines as a custom connector, with no install: https://aaj-engines-mcp.sarojkjha10.workers.dev/mcp