A/B Test Significance Calculator (Frequentist + Bayesian)

When to use it

Before launching a test (to plan sample size and duration), during a test (to decide whether to peek), and at the end (to call a winner without fooling yourself).

What you'll need

• Visitors and conversions per variant (control + 1–4 challengers)
• Baseline conversion rate (for the planner)
• Minimum detectable effect you actually care about

How the math works

Frequentist: two-proportion z-test with continuity correction, plus 95% confidence intervals on the absolute and relative lift. Bayesian: Beta-Binomial conjugate model with a uniform prior, Monte-Carlo sampled to compute each variant's probability of being best and the expected loss of switching. Sample-size planner uses standard power calculations (α = 0.05, 1−β = 0.8 by default, configurable).

How to read the output

For a confident call, the frequentist p-value should be below 0.05 AND the Bayesian probability-to-be-best should exceed 95% AND expected loss should be near zero. Single signals can mislead — peeking issues, novelty effects, and seasonality all bite. The planner tells you the minimum runtime to avoid each.

Related tools & playbooks

Pair this with the B2B Website & CRO Playbook and the A/B Testing Complete Guide for the full methodology.

FAQ

Frequentist or Bayesian — which should I use?

Use both. Frequentist gives you regulatory-style rigor (a p-value), Bayesian gives you a decision-making metric (probability the change is better) and lets you peek without inflating false positives.

How big should my sample be?

The planner returns the exact number, but typical B2B SaaS tests need 5–15× more traffic than founders expect. Tests on low-traffic pages are usually unwinnable on statistics alone.

Can I stop a test early if it's winning?

Only with Bayesian guardrails and a pre-committed expected-loss threshold. Stopping a frequentist test early dramatically inflates false-positive rates.