Incrementality Testing for Startups: Prove Your Ads Actually Work

1. Blog
2. Measurement & Budget

By Saroj Jha · July 17, 2026 · 13 min read

Incrementality testing measures the sales your marketing actually caused, not just touched. How geo lift tests work, a worked example, the full test menu, and how to run your first test without an enterprise vendor.

From this article

Only one of those two numbers belongs in unit economics.

That distinction, caused versus touched, is where startup budgets quietly leak, because every ad platform grades its own homework. Branded search shows it most starkly. When eBay switched off its brand-keyword ads, shoppers clicked the free organic link instead, and the ads showed no measurable short-term benefit^[1].

Platform-reported conversions and incremental conversions are different quantities. Only the second describes what your marketing caused, so only the second belongs in CAC and payback. To measure it, hold the spend back from a matched group and compare.

Why this went from enterprise luxury to table stakes

Three forces converged. Signal loss came first: Apple’s tracking prompt, cookie blocking in Safari and Firefox, and link-stripping in Mail, Messages and Safari Private Browsing^[5] made user-level attribution steadily less reliable. Platform inflation was always there but became undeniable: when every channel’s dashboard claims credit for the same conversion, adding up platform-reported revenue routinely “proves” you earned more than you actually did. Across 15 large Facebook experiments, the observational methods advertisers usually rely on often missed the true effect, even with rich user data^[6]. And CFO scrutiny closed the loop: “the platform says ROAS is 4” stopped surviving board meetings.

By July 2025, 52% of US brand and agency marketers said they used incrementality tests or experiments to measure campaigns^[2]. Conveniently, the workhorse method needs no vendor, no user tracking, and no data science team; Meta’s GeoLift and Google’s Meridian GeoX are free open-source code for it^[7],[8]. It needs geography, discipline, and 4–8 weeks of patience.

How a geo lift test actually works

The workhorse design is the geo holdout: divide your market into geographic units (US DMAs, or regions elsewhere), match them on baseline performance, then hold back 20–30% of them as a control while the rest keep (or change) the treatment. After 4–8 weeks, compare the change in test geos against the change in control geos. That difference-in-differences read cancels out the seasonality and market shifts both groups share^[9].

Three design details do most of the quality work. Matching: pair geos on baseline conversion volume and trend before randomizing. Pre-period: you want clean history so you know each geo’s normal behavior. Unit choice: bigger units (states, DMAs) are cleaner; smaller units (cities) give you more statistical material but leak more. In my view, startups land on 15–25 usable geos.

Two directions. Turn it off (holdout): pause a channel in control geos. If conversions there barely dip, you found budget to reclaim. Turn it up (scale test): boost spend 30–50% in test geos. Did conversions rise proportionally, or have you hit diminishing returns?

The math, worked

Running case: Example Co. Fictional

Example Co., a B2B SaaS startup, spends $3,000 a week on Meta prospecting across its test geos. It matches 20 geos, holds out 6 (30%) as control, and pauses Meta there for six weeks.

Pre-period: test geos 300 signups a week, control geos 200. During the test: test geos 330 (+10%), control geos 196 (−2%). The test geos rose with nothing changed, so the market was rising; a before-and-after look at the paused geos alone would say Meta was worth just 4 signups a week. Every number here is invented for the walk-through; substitute your own.

Difference-in-differences: +10% − (−2%) = 12 points of lift. In absolute terms: 300 × 12% = 36 incremental signups per week that exist because Meta ran. Cost per incremental signup = $3,000 ÷ 36 ≈ $83. Meanwhile the platform claimed 90 conversions, a platform-reported CAC of $33. If a signup is worth $250 in expected revenue, the incremental ROAS is 3.0 (36 × $250 ÷ $3,000), against the 7.5 the platform’s count implies. Both numbers are “true” in their own universe; only one belongs in your unit economics.

And 36 is a best estimate, not a fact. Chance alone puts the 95% range at about 4 to 65 a week, or $46 to $713 per signup, so report the range with the number.

What makes a test trustworthy

Pre-register everything. Hypothesis, geos, duration, primary metric, and the decision you’ll make at each outcome, written down and locked before launch. Power it honestly: name the smallest lift worth acting on (the minimum detectable effect, or MDE), then check your volume can see it: at 200 conversions a week per arm, a six-week test can’t reliably detect much under 18%. Freeze the machine: bids, creative, targeting and landing pages stay untouched for the duration. Mid-test “optimizations” are how teams spend six weeks learning nothing.

The test menu (geo isn’t the only option)

Geo holdout: the workhorse, and the strongest read you can run yourself, with no tracking required. Geo scale-up: the diminishing-returns detector; run it before believing any “pour more into the winner” plan. Time-based on/off: switch the channel on and off in alternating periods; weaker, since anything else that changes those weeks gets mixed in, but workable when customers sit in a few markets. Platform lift studies: the platform randomly holds users or regions back, made cheap by ghost ads, which log the ads a held-out user would have seen without showing them^[10]. Sound, but they see only that platform’s ads and tracked conversions, and access is gated: Google’s user-based Conversion Lift needs a $5,000 campaign budget and 1,000 observed conversions^[4], and Meta sends advertisers to a Meta representative^[11]. Branded-search holdout: usually your best first test: cheap, fast, and aimed at the likeliest padding. First check whether rivals bid on your name; on Bing, they took 18–42% of clicks from brands that didn’t defend it^[12].

Reading the result, and acting on it

Decide the decisions in advance. Lift ≥ MDE and cost per incremental conversion ≤ target → scale ~20% and retest in one to two quarters. Lift below MDE → reallocate 15–25% of budget, iterate creative or targeting, retest. Branded search shows most conversions persist organically → reclaim budget without ceremony. Test disagrees with your model → the test wins, every time. This test-then-recalibrate loop is how MMM-Lite and incrementality work as a system; Google’s open-source Meridian model turns lift results into calibrated priors^[13].

When a startup actually needs this

Not at $5K/month across two channels: at that size, your budget is the experiment. The test earns its overhead when a channel is taking a meaningful share of budget on faith, you’re about to scale a “winner” only a platform’s own attribution has vouched for, or a model told you something expensive and you’d like a second opinion. In my view, one test per quarter on your most consequential channel is all the cadence a Seed–Series B company needs.

Your first test, concretely

Do this

Pick the channel you’d most hate to be wrong about; for most startups, that means branded search first. Write the one-page pre-registration, match and split your geos, hold out 20–30%, run 4–8 weeks untouched, read the diff-in-diff with its range, and do the thing you committed to.

Free template

Pre-register your first test before it starts

The free MMM-Lite Starter Kit includes an incrementality-test pre-registration template and the decision rules, ready to lock before launch.

For the demand that never reaches your attribution at all, see my piece on dark social.