Audit of Bayesian stopping rules

Simple terms

A “95% chance to beat control” label can look intuitive while answering a narrower question than merchants think. We simulated common published rules to measure what happens when users check them daily.

The setup

The audit reimplemented three rules from public methodology descriptions. It did not run or inspect any vendor’s proprietary production code, and the results should be read as an audit of the published rules and parameter choices—not as a claim about every implementation or product safeguard.

• Chance to beat at least 95%: a flat-prior posterior probability that the treatment beats control.
• Expected-loss stopping at 1% relative: stop when the estimated loss from shipping the current leader falls below a conservative threshold.
• Expected-loss stopping at 5% relative: the same rule with a more permissive threshold.

Each rule was evaluated daily for 28 days under realistic small-business traffic and conversion rates. The run included 360,000 A/A experiments plus 60,000 simulations with a real winner, using fixed seeds.

In an A/A experiment the two variants are identical. Every declared superior winner is therefore false by construction.

Results

Rule	False-winner rate in A/A	Median declaration day
Chance to beat at least 95%	40%–46%	day 4–6
Expected loss, 1% relative	57%–99.8%	day 4–9
Expected loss, 5% relative	about 100%	day 1–2

The chance-to-beat rule declared a false winner in nearly half of the A/A experiments within a month of daily peeking, usually during the first week.

Why chance-to-beat inflates

With a flat prior, the probability that B exceeds A behaves similarly to a one-sided significance statistic. Repeatedly checking it and stopping at the first threshold crossing creates an optional-stopping problem.

Bayesian arithmetic does not automatically make every stopping rule safe. The posterior calculation can be correct while the user-facing decision procedure fails to preserve the interpretation attached to the threshold.

Why expected loss eventually fires

Expected-loss stopping answers a different question: “How much could we lose by shipping the current leader?”

Under a true tie, the expected loss of choosing either arm approaches zero as more data arrives. The rule is therefore designed to conclude, not to prove superiority. Declaring equivalence or indifference can be reasonable; presenting the selected arm as a superior winner with an observed lift creates a false lesson.

This also explains the counterintuitive traffic result: the 1% expected-loss rule produced more winner declarations at higher traffic because the estimated loss shrank faster.

Speed and error together

The audit compared a real 10% lift scenario using matched seeds:

Measure	Chance to beat at least 95%	Apex sequential verdict
Median visitors to detect the winner	about 18,000	about 86,000
Wrong-winner rate	4.6%	about 0.04%
False winners in A/A with daily peeking	40%–46%	0.2%–3.9% against a 5% budget

The chance-to-beat rule was roughly 4.8 times faster in that scenario, but the error behavior was not comparable. A useful speed claim must always be paired with the risk spent to obtain it.

Apex’s own engine has the opposite limitation: it often spends less than its allowed 5% error budget and can take longer than necessary. That finding is public in the Apex methodology and certification. The engineering objective is faster decisions at a disclosed risk level, not a contest between Bayesian and frequentist labels.

Questions to ask any experimentation platform

1. What false-winner rate does the decision rule produce on A/A tests under daily peeking?
2. Does the dashboard number keep the same meaning when it is inspected repeatedly?
3. Is an early-stopped lift corrected or clearly labeled for winner’s-curse bias?
4. Does an expected-loss stop mean superiority, equivalence, or simply that either decision is cheap?
5. Can the platform show a deterministic audit with stated seeds and parameter choices?

Interpretation limits

These results apply to the documented rules, flat-prior assumptions, tested thresholds, and simulated traffic grid. A production product can add safeguards, different priors, minimum sample rules, or other decision policies that change the outcome.

The result is not “Bayesian statistics are bad.” The result is that a posterior probability and a stopping policy are separate things, and the interface must state honestly what decision guarantee the combination provides.