Your A/B test winner is about half as good as it looked — measured on 32,363 real randomised experiments, not simulated ones.
▶ Try the live demo — pick a real experiment, see which headline you would have shipped, then watch it get scored on traffic it never saw.
- The problem — and why it matters
- The data
- 1. The winner's curse, measured
- 2. Shrinkage fixes the estimate, not the choice
- 3. Peeking, priced on real traffic
- 4. Every extra arm buys you a fake winner
- 5. Why all of this happens: nobody had the traffic
- The interface
- Key design decisions
- Limitations
- Reproduce it
- Project layout
Run four headlines against each other, ship the one with the highest click-through rate, and report the lift. Everyone does this. The catch is that you did not pick the best headline — you picked the headline with the best estimate, and the noise that made it look good does not ship with it.
That is the winner's curse, and it is usually argued about with simulations. This project measures it instead. Upworthy ran tens of thousands of genuinely randomised headline tests between 2013 and 2015 and released the results, so the truth can be checked directly: deal each arm's impressions at random into a selection half and a held-out half, pick the winner on one, score it on the other. Because assignment within an arm is exchangeable, that split is exactly what a smaller sample would have looked like. No effect sizes are simulated anywhere in this repository — every number below is real clicks.
The headline results:
| measured on the archive | |
|---|---|
| Lift the chosen arm appears to have | +31.9% |
| Lift it actually has, on held-out traffic | +14.2% |
| …and if a significance test blessed it | +51.6% → +27.2% |
| Same arm wins again on held-out traffic | 43.5% of the time |
| False positives after peeking 10 times at a null test | 17.9% (nominal 5%) |
| A "significant" winner among 6 identical arms | 18.2% of tests |
| Smallest lift the median test could actually detect | +68%, against a real median of +12% |
The Upworthy Research Archive (Matias, Munger, Le Quere & Ebersole; CC BY 4.0) records every package Upworthy tested: a headline and image shown to a random slice of traffic, with impressions and clicks. Packages sharing a test id are the arms of one experiment.
After dropping tests containing an arm with under 1,000 impressions (409 rows) and the long tail above 12 arms (384 rows), the corpus is:
32,363 experiments · 150,024 arms · 537M impressions · 121 test-weeks · 1.52% pooled CTR · 3,120 median impressions per arm
Most tests ran four to six arms on about three thousand impressions each. Hold on to that number — it is the whole story.
Just under half the tests varied the headline (15,672), a third varied the image (9,343), 2,059 varied both, and 5,289 varied neither — only the share text.
Pick the best arm on half the traffic; score that same arm on the other half.
Left: the same 32k experiments, scored twice. Middle: the more arms you test, the more of the winner's lift is imaginary. Right: how often the pick survives a second look, against the rate you would get by choosing at random.
- The winner's lift over its test average: +31.9% apparent, +14.2% real. 55% of it evaporates.
- Restrict to winners a two-proportion z-test called significant at 5% — 39.3% of all decisions — and the numbers get bigger but no more honest: +51.6% apparent, +27.2% real. Significance helps; it still leaves you overstating by nearly a factor of two.
- The chosen arm wins again on held-out traffic 43.5% of the time. That beats chance (25% for a four-arm test), but at ten arms it is 24.3% against a chance rate of 10%.
- Selecting on more traffic shrinks the illusion without removing it: picking on 30% of impressions loses 67% of the apparent lift, on 50% loses 55%, on 70% loses 47%.
The gradient by arm count is the mechanism in one line: at 2 arms the apparent lift is +11.1% and the real one +1.0%; at 10 arms, +57.2% and +20.6%. Every additional arm is another draw from the noise distribution, and the maximum of more draws is larger.
Tests that varied the headline produced the largest real effects (+17.1% held-out), images +10.8%, and tests that varied only the share text +3.4% — a sanity check that the split-half method finds real signal where you would expect it and near-nothing where you would not.
Fit a beta-binomial prior across the arms within each test — one concentration parameter κ for the whole archive, fitted by maximum likelihood on the selection half only, since that is all an analyst has at the decision point. The fit gives κ ≈ 2,318, implying real between-arm variation of about 0.23 percentage points on a 1.5% base: real, but small next to the ±0.2pp of noise on 3,000 impressions.
Three ways to answer "how good is the arm we just picked?", scored against held-out traffic.
| estimator of the winner's true CTR | relative bias | RMSE | MAE |
|---|---|---|---|
| observed CTR of the winner | +13.0% | 0.00309 | 0.00255 |
| empirical-Bayes posterior mean | −2.7% | 0.00293 | 0.00195 |
| the test average (ignore the winner) | −13.5% | 0.00471 | 0.00300 |
Shrinkage removes about 80% of the bias and 24% of the absolute error. What it does not do is change which arm you ship: when arms have equal traffic, the posterior mean is monotone in the raw rate, so the ranking is untouched. The cure for the winner's curse is a better number, not a better pick — and the useful part of that number is that it stops you promising +32% to your stakeholders when +14% is coming.
To price repeated testing honestly you need a null that is real rather than assumed. Take one real arm and deal its impressions into two pseudo-arms: the true difference is exactly zero by construction, but the clicks are real clicks at a real rate.
Left: 150,024 real A/A splits. Right: the same rules applied to real arm pairs, with the declared direction checked against traffic the stopping rule never saw.
| looks | fixed horizon | peek every look | O'Brien-Fleming | Pocock | always-valid (mSPRT) |
|---|---|---|---|---|---|
| 1 | 4.9% | 4.9% | 4.9% | 4.9% | 0.03% |
| 2 | 5.0% | 8.2% | 4.9% | 4.6% | 0.03% |
| 5 | 4.9% | 13.6% | 4.8% | 4.3% | 0.05% |
| 10 | 5.0% | 17.9% | 4.9% | 3.9% | 0.06% |
| 20 | 5.0% | 21.5% | 4.7% | 3.6% | 0.06% |
The fixed-horizon test lands on 4.9–5.0% no matter how much traffic accumulated first — the machinery is calibrated on real data, which is what makes the other columns believable. And when a peeking analyst does stop early on a test where nothing is happening, the effect they report is not marginal: the median false lift at the stopping point is +114% at ten looks.
The boundaries are computed here rather than imported — the Armitage-McPherson recursion, propagating the sub-density of the test statistic look by look with an FFT. It reproduces the published tables (Pocock at 5 looks: 2.413; O'Brien-Fleming at 4: 4.049 / 2.863 / 2.337 / 2.024), which is what the tests pin.
Why it works: O'Brien-Fleming spends almost none of the error budget early, so the tenth look still costs about the same as a single fixed test.
The part worth internalising is the price. Watching one randomly chosen pair of arms from each of the 32,363 tests through ten looks, then checking each declaration against held-out traffic:
| rule | declares a winner | direction holds up | shrinkage of the declared lift |
|---|---|---|---|
| wait for the end | 17.1% | 87.1% | 42% |
| peek every look | 30.4% | 77.5% | 71% |
| O'Brien-Fleming | 16.4% | 87.0% | 50% |
| Pocock | 13.1% | 87.1% | 57% |
| always-valid (mSPRT, τ fitted) | 1.6% | 96.8% | 37% |
Naive peeking finds 78% more winners and they are markedly worse: it gives up 9.6 points of replication and reports lifts that are 71% noise. O'Brien-Fleming costs 0.7 percentage points of declarations against waiting for the end — you may look ten times, stop early on average at 75% of the horizon, and keep both your error rate and your replication rate. That is close to free.
The always-valid mSPRT is the interesting outlier. Priced with a mixture width τ fitted from the archive's own between-arm variation (τ = 0.0036), it is far more conservative than its 5% level suggests, because τ is smaller than the standard error of the thing being measured. Widening it trades that back:
| τ | false positives (A/A) | declarations on real pairs |
|---|---|---|
| 0.001 | 0.0% | 0.01% |
| 0.005 | 0.16% | 3.0% |
| 0.010 | 0.44% | 5.2% |
| 0.020 | 0.57% | 5.3% |
Always-valid inference is not free power — it is power spent on the effect sizes you told it to expect.
Same construction, more pseudo-arms: deal one real arm's impressions into k equal parts, then ask what a dashboard would ask — is the best of them significantly better than the rest?
Left: the true difference between these arms is zero. Right: how good the best of k identical arms looks anyway.
| arms | "significant" winner | median apparent lift of the best arm |
|---|---|---|
| 2 | 5.05% | +7.7% |
| 4 | 11.4% | +22.5% |
| 6 | 18.2% | +35.3% |
| 10 | 31.1% | +57.4% |
At two arms the procedure lands exactly on its nominal 5%, which is the control that makes the rest of the column mean something. Upworthy's typical test ran four to six arms; on a test where every headline is equally good, that design declares a significant winner about one time in six, and the winner it declares looks 20–35% better than average.
None of the above is anyone's mistake. It is arithmetic.
Left: the effect each test could have detected, against the effect that was actually there. Right: the gap by traffic decile.
At a 1.3% baseline and 3,089 impressions per arm, the smallest lift a two-arm test could detect with 80% power is +68%. The median real lift of the best arm is +12%. So:
- 4.6% of tests were adequately powered for the effect they contained.
- Median realised power: 8.8%.
- Detecting the median real effect would need 86,101 impressions per arm — 28× the traffic these tests actually got.
Which is why the winner's curse here is so large: when power is 9%, a "significant" result is mostly a large error, and the arm carrying the largest error is exactly the arm you ship.
One aside for anyone using an online calculator: the textbook MDE formula evaluates the variance at the baseline rather than at the alternative, and at these rates it says +59% where the exact inversion of the power function says +68%. This repo solves for the MDE numerically and reports both.
A single self-contained HTML file (no build step, no CDN, light and dark) served by FastAPI, and deployed as a static Space by shipping the same file next to its JSON payload. The split runs in your browser — a seeded PRNG and an exact multivariate-hypergeometric deal, ported from the Python — so "re-split the traffic" really re-deals the impressions and the verdict can change in front of you.
A real test: the arm you would have shipped claimed +72%; on traffic it never saw it was 41% worse than average, and a different headline won.
And one where the winner is real: +34.8% claimed, +44.3% delivered. The z-path shows a peeking analyst stopping at look 4 — right call, wrong reason, since the same path dips back inside the boundary twice.
make serve # http://127.0.0.1:8000
curl localhost:8000/healthz
curl "localhost:8000/sample?index=6" # one experiment, split, scored, with its z-path
curl localhost:8000/api/summary # every finding as JSONSplit the traffic, do not model it. The alternative to sample splitting is assuming a prior over true effects and computing the expected shrinkage. That answer would be exactly as good as the assumption. Dealing real impressions into halves needs no assumption beyond exchangeability within an arm, which randomisation guarantees.
Every null is built from real data. The A/A splits and the k-arm null tests are real Upworthy traffic re-dealt, not Bernoulli draws. That is why the calibration checks (fixed-horizon test at 4.9%, two-arm selection at 5.05%) are meaningful: they say the pipeline is measuring what it claims to measure.
Pool over decisions, not over tests. An earlier version averaged each test's lift across replicate splits and then filtered to the significant ones. That quietly changes the subgroup as the replicate count rises — the "significant winners" headline moved from 62% to 83% shrinkage between 3 and 20 replicates, which is a property of the aggregation, not the archive. Conditioning on the decision instead gives 47%, stable to three decimals from 5 replicates to 20.
Fit the prior on the selection half only. κ is estimated from the same half of the traffic used to pick the winner, because that is the information an analyst actually has. Fitting it on everything would leak the answer.
Solve for the MDE instead of quoting the formula. Inverting
power_two_proportion numerically keeps power and MDE mutually consistent —
there is a test asserting a test run at exactly its MDE has exactly 80% power.
Write the sequential machinery out. The alpha-spending boundaries are a numerical recursion in this repo rather than a dependency, so the tests can pin them against published tables and the demo can draw them in the browser.
- Split-half selection is harsher than the real thing. The winner is chosen on half the traffic, so the curse measured at frac = 0.5 is larger than what Upworthy's analysts faced at full n. The sensitivity analysis brackets it: 67% of the lift is illusory when selecting on 30% of traffic, 47% at 70%. The true full-sample figure is below 47%, and the direction of the bias is known.
- "True" means held-out, not infinite. The evaluation half is itself finite (~1,500 impressions per arm), so it is unbiased but noisy. That inflates RMSE for every estimator equally and leaves the bias comparisons clean, but a held-out lift of +14% carries real uncertainty.
- Random splitting assumes a stable rate. Dealing impressions at random models a test accumulating traffic under a constant CTR. It does not model day-of-week effects, novelty decay, or a headline that ages — real complications the peeking analysis therefore understates.
- These are 2013–2015 clickbait headlines at a ~1.5% CTR. The mechanism generalises; the magnitudes are a function of that baseline and that traffic. A test at a 20% conversion rate on 100k users per arm is a different world — the point is to run the same split on your data, not to import these percentages.
- The beta-binomial prior is shared across the whole archive. One κ for 132 weeks of tests is a strong assumption; per-week or per-topic priors would fit better and would need more care about leakage.
- Upworthy's own
winnerandsignificancefields are ignored. Their internal rule is undocumented, so every "winner" here is one this pipeline picked with a stated rule.
git clone https://github.com/UsmarHaider/overlift && cd overlift
make venv # python3 -m venv .venv && pip install -e ".[dev]"
make test # 106 tests, fully offline -- no archive needed
make all # download (95 MB) -> analyse -> figures -> web payload
make serve # the interface on :8000make all takes about a minute of compute once the archive is downloaded (OSF is
slow; the download script resumes). Everything is seeded: SEED = 20260812 in
src/overlift/config.py. Reports land in reports/*.json, figures in
docs/figures/, and the web payload in artifacts/web/data.json.
The test suite builds its own corpora with known ground truth — one where every arm is genuinely identical and one where a single arm is genuinely 40% better — and asserts the pipeline finds a curse in the first, the real effect in the second, and its own published boundary tables in between. CI never downloads the archive.
To publish the demo: HF_TOKEN=... python scripts/deploy_space.py.
src/overlift/
config.py paths, seed, thresholds, palette
data.py archive CSVs -> tidy (test, arm) table + filter report
stats.py proportion tests, exact MDE/power, beta-binomial shrinkage
splitting.py exact re-dealing of real impressions into sub-samples
boundaries.py alpha-spending boundaries (Armitage-McPherson) + mSPRT
curse.py analysis 1-2: selection, held-out scoring, estimator bias
peeking.py analysis 3: A/A nulls, stopping rules, declaration quality
multiplicity.py analysis 4: k identical arms
power.py analysis 5: MDE vs the effects that were really there
figures.py the seven README figures
sample.py the web payload: real tests, replayable
service.py FastAPI app serving the page and a read-only API
ui/index.html the entire interface, self-contained
scripts/ download_data.py, screenshot.py, deploy_space.py
tests/ 106 offline tests
reports/ every number in this README, as JSON
Data: Upworthy Research Archive — J. Nathan Matias, Kevin Munger, Marianne Aubin Le Quere, Charles Ebersole (2021), CC BY 4.0, https://osf.io/jd64p/. The archive is downloaded by script and never redistributed here; the demo ships a 240-test sample with attribution.
Method references: O'Brien & Fleming (1979); Pocock (1977); Lan & DeMets (1983) for alpha spending; Johari, Koomen, Pekelis & Walsh (2017) for always-valid inference; Jennison & Turnbull for the boundary tables the tests pin against.
MIT licensed.