Your winning ad might be a coin flip (check before you scale)

Thursday's creative review. The new hook — Ad B — converted at 1.80%. The old one, Ad A, at 1.50%. That's a 20% lift. Budget moves to B, A gets paused, and "new hook wins" goes into the learnings doc.

A month later, B is converting at 1.53% — right where A always sat. The hook didn't stop working. It never worked better. One week and 8,000 visitors per ad simply can't separate a 20% gap from luck.

Two identical ads, one "winner"

The simulator below runs an A/A test: both ads convert at exactly 2%, so any gap is pure chance. Re-run it a few times at low traffic, then drag the slider up.

At a few hundred visitors a day, 10–25% gaps between identical ads are routine. At several thousand a day they shrink to a few percent. Nothing about the ads changed — only the amount of data. The smaller the test, the bigger the lies it can tell.

Why platforms crown winners too early

• Uneven delivery isn't a test. Put four ads in one ad set and the platform picks a favourite within a day or two and starves the rest. The "losers" never got enough traffic to lose fairly.
• Peeking. Checking daily and stopping the first time it looks significant gives luck many chances to win. Commonly cited simulations put the false-winner rate at 20–30% instead of the 5% you think you're accepting.
• Too many variants. Test ten ads against a control and, by chance alone, one will usually look like a winner at 90% confidence. More arms need more data or a stricter bar.

How much data does a test need?

Two things decide it: how often people convert, and how small a difference you want to detect. Low conversion rates and small lifts need a lot of traffic. Try your own numbers:

Under the hood is a standard rule of thumb for 95% confidence and 80% power: n ≈ 16 × p(1−p) ÷ (p × lift)² visitors per ad. The version worth memorising: to detect a 20% lift you need roughly 400 conversions per ad. A 50% lift needs only about 60; a 10% lift needs around 1,600. That's why bold ideas are far easier to test than small tweaks.

Reading the result honestly

When the test reaches its planned size, check significance once. This query returns the lift, a z-score and a 95% range for the true difference:

WITH totals AS (
  SELECT
    SUM(IF(arm = 'A', visitors, 0))    AS n_a,
    SUM(IF(arm = 'A', conversions, 0)) AS c_a,
    SUM(IF(arm = 'B', visitors, 0))    AS n_b,
    SUM(IF(arm = 'B', conversions, 0)) AS c_b
  FROM creative_test_daily
  WHERE test_name = 'new_hook_nov'
    AND day BETWEEN '2026-11-02' AND '2026-11-15'   -- window fixed before launch
),
rates AS (
  SELECT c_a / n_a AS p_a, c_b / n_b AS p_b,
         (c_a + c_b) / (n_a + n_b) AS p_all, n_a, n_b
  FROM totals
)
SELECT
  ROUND(p_a * 100, 2)                                                    AS cvr_a_pct,
  ROUND(p_b * 100, 2)                                                    AS cvr_b_pct,
  ROUND((p_b / p_a - 1) * 100, 1)                                        AS lift_pct,
  ROUND((p_b - p_a) / SQRT(p_all * (1 - p_all) * (1 / n_a + 1 / n_b)), 2) AS z,
  -- 95% range for the true difference, in percentage points
  ROUND((p_b - p_a - 1.96 * SQRT(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)) * 100, 2) AS low_pp,
  ROUND((p_b - p_a + 1.96 * SQRT(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)) * 100, 2) AS high_pp
FROM rates;

A z beyond ±1.96 clears the 95% bar. The range is more telling: if low_pp is below zero, the true effect might be nothing — or negative. For the Thursday "winner" above, z is about 1.49 and the range runs from roughly −0.10 to +0.70 points. It includes zero, so the test proved nothing.

A test protocol that holds up

1. Write down the question and one metricOne hypothesis, one primary metric — conversion rate, CPA or CTR — chosen before launch.
2. Size it before you startThe lift worth catching sets the traffic and days. If it's too long, test a bolder idea.
3. Split traffic properlyUse the platform's A/B or experiment feature, with two or three variants at most.
4. Don't peek, don't stop earlyRun to the planned size plus the lag window. Daily checks are for broken links, not winners.
5. Read once, record the rangeLog the lift, the confidence and the range. "Not proven" is a valid outcome — it stops you scaling luck.

• No "winner" under ~400 conversions per arm for a 20% lift — smaller tests are directional, never proven.
• Test big swings, not button colours — new angles and formats produce lifts you can measure on normal budgets.
• Re-check winners after scaling — compare four weeks at full budget against the test. If the lift vanished, update the doc.

Quick gut-check