Holdout Test Planner

A free tool for anyone asking whether a channel's spend caused conversions or only came before them. Attribution cannot tell you; a holdout can. Plan the test before you spend, then read it out honestly, including whether it was ever big enough to answer the question. It runs the engine of the incrementality-and-mmm agent skill in your browser; nothing you enter is sent anywhere.

Plan the test

Give the baseline conversion rate, the smallest lift you would act on, the daily traffic and the share held out as control, and pick the confidence (90, 95 or 99%) and power (80, 90 or 95%). The engine gives the sample per group from the standard two-proportion formula and the days your traffic takes; the smaller group sets the pace. With a value per conversion, it also gives what the holdout costs: the extra conversions the control group gives up at that lift. Under 7 days, run a full week anyway. Over 42 days, drift and competing changes will contaminate the result; accept a larger lift or hold out whole markets instead.

Read the test

Give each group's size and conversions, and optionally the spend, the value per conversion and the lift you would act on. You get the relative lift, the p-value, the interval on the difference, incremental conversions against what the treatment group would have got at the control's rate, incremental CAC and iROAS. The verdict is incremental, negative, not significant, or inconclusive. Inconclusive means the test could only have detected a lift bigger than the one you care about: it was too small to answer, which is not the same as the channel not working.

Why I built it

I have watched a holdout that was never big enough get reported as "the channel doesn't work", and budget move on the strength of it. The number that matters most is the one most tools leave out: the smallest lift the test could have found. So this planner puts it next to the verdict.

Limits

It assumes people are split at random. A geo test holds out whole markets, which differ more than people do; the engine does not model that, so a geo test needs more than this sample, with markets matched on baseline rate and volume.
Hold pricing, creative and other channels constant for the whole window, and run to the planned length. Stopping when the result looks good inflates false positives.
It designs and reads one holdout. It is not a media mix model.

For an on-site test of a page or flow, use the A/B Test Significance Calculator. For which ad wins inside a channel, use the Ad Creative Test Calculator.

Run this in your agent

The free AAJ agent skill Incrementality & MMM does this job inside a coding agent, on your own data. Install it with npx skills add sarojkjha/aaj-marketing-skills --skill incrementality-and-mmm.

Using Claude or ChatGPT instead? Add the AAJ engines as a custom connector, with no install: https://aaj-engines-mcp.sarojkjha10.workers.dev/mcp

Frequently asked questions

What is a holdout test?

You keep a random group of people from seeing a channel's spend and compare their conversions with the group that did. The difference is what the spend caused. Attribution cannot tell you that, because it only sees the world where the ad ran.

How big does a holdout test need to be?

It depends on your baseline conversion rate, the smallest lift you would act on, the confidence and the power. Small lifts on low rates need very large groups. Enter your numbers and the planner gives the sample per group and the days.

My test was not significant. Does the channel not work?

Not necessarily. If the test could only detect lifts bigger than the one you care about, it was too small to answer, and the planner says inconclusive. If it was big enough and found nothing, that is a real finding: treat the channel's attributed conversions as suspect.

Is my data stored or shared?

No. The planner runs in your browser and nothing you enter is sent to AAJ or kept after you close the page. If you copy a link to your results, your numbers travel inside that link, so anyone you send it to can see them.