Ad Creative Test Calculator

A free tool for anyone testing ad creative on Meta, LinkedIn, Google, TikTok or any paid channel. Size the test before it runs, so you know whether your traffic can answer the question. Then read it out on the metric that pays, and get "no winner" when that is the honest answer. It runs the engine of the ad-creative-testing agent skill in your browser; nothing you enter is sent anywhere.

Before the test: how long to run it

Give the control's current rate on the metric you judge by (conversions per impression by default, or click-through rate, or conversion rate from click), the smallest lift worth detecting, the number of variants and the daily impressions each one gets. The engine works out the impressions each variant needs for a two-sided test at your significance level and power, and the days your traffic takes to get there. With more than one challenger it splits the significance level across the comparisons (Bonferroni). If the test needs longer than you allow (28 days by the engine's default), it says what lift your traffic can detect in that time and what to change: fewer variants, a bigger idea, or judging on clicks first.

After the test: is the winner real?

Give each variant's impressions, clicks, conversions and spend, control first. You get each variant's rate with a 95% interval, its lift on the control, the p-value against the corrected threshold and the cost per conversion, then a verdict: a winner, the control wins, no winner yet, or not a clean read. The engine will not name a winner when one variant got more than twice another's impressions, an arm has fewer than 10 events, the smallest arm is short of the planned size, or a click-through winner costs more per conversion than the control. These thresholds are the engine's rules, not industry benchmarks.

Why I built it

I keep seeing ads scaled on three days of data and a better click-through rate. The ads reached different people, the test stopped at a lucky moment, and clicks were never the result being paid for. I wanted a calculator that says "no winner yet" out loud, and says why.

Limits

• It reads the numbers you give it. It cannot see whether your ad platform split the audience; use the platform's experiment or A/B test feature where it has one.
• Read the test once, at the planned size. Looking every day and stopping when something looks good inflates false winners.
• "No winner" means the change did not matter enough to detect at this sample. Test a bigger idea next, not the same idea for longer.

For a landing page or email test, use the A/B Test Significance Calculator. To find out whether the channel works at all, use the Holdout Test Planner. For the channels themselves, read the Digital Advertising Playbook.

Run this in your agent

The free AAJ agent skill Ad Creative Testing does this job inside a coding agent, on your own data. Install it with npx skills add sarojkjha/aaj-marketing-skills --skill ad-creative-testing.

Using Claude or ChatGPT instead? Add the AAJ engines as a custom connector, with no install: https://aaj-engines-mcp.sarojkjha10.workers.dev/mcp