How Apex statistics work

Simple terms

Apex is built to reach useful business decisions quickly without presenting noise as a winner. We measure whether that promise holds instead of asking customers to trust a label.

Statistics as risk management

A/B testing is not science for its own sake. It is a way to manage the cost of a wrong decision against the cost of waiting.

The familiar 95% convention is not a universal quality grade. A reversible copy change, a checkout change, and an automated harm pause carry different consequences. In principle, the correct threshold comes from the economics of the decision: reversibility, downside, traffic cost, and time to learn.

Apex currently keeps its public decision contract simple:

Decision	Shipped threshold	Reason
New-experiment winner default	80%	Out-of-box shop default; configurable per experiment from 80% to 99%
Guardian harm action	99%	A false pause is client-facing and should be rare

Lower thresholds can be rational for cheap, reversible decisions, and stricter thresholds can be rational for expensive ones. Apex currently exposes a configurable winner threshold, not automatic risk tiers based on the kind of change. New experiments inherit the shop's testing defaults, whose out-of-box winner threshold is 80%. Any future automatic threshold policy must be measured against the same certification harness before it becomes part of the product contract.

The 5% false-positive budget in the certification results below applies to the 95% winner-threshold configuration used in that registered grid. It does not turn the shipped 80% default into a 95% contract.

One-sided decisions

A store normally asks a directional question: “Should we ship B because it is better than A?” The matching test is one-sided. A two-sided test asks whether B differs in either direction and spends additional sample on a question that does not match the shipping decision.

The direction is part of the disclosed decision rule. It is not free speed: choosing a direction is itself a risk decision, which is why high-stakes safety actions use a stricter threshold.

Repeated peeking

A classical fixed-horizon test assumes one evaluation at its planned sample size. If a dashboard checks that test every day and acts the first time it becomes significant, the false-positive rate no longer matches the label.

Apex measured this behavior directly under daily peeking:

• At the registered 95% winner threshold, the live sequential winner engine conformed in all 24 certified scenarios, with measured false-positive rates from 0.2% to 3.9% against a 5% budget. The grid included a 90-day horizon and three-arm experiments.
• A classical test evaluated daily violated its budget in all six scenarios, measuring 27.1% to 27.6% false positives against a 5% label.
• The former revenue significance path measured 21.384% to 27.284% false positives against 5%. Apex gated that inference instead of continuing to display a confidence claim that failed its audit.

This is why the live verdict uses an always-valid sequential method and why fixed-horizon statistics must not be presented as safe to inspect continuously.

Speed is part of validity

The certification also found that the live sequential engine is conservative. Across scenarios it often spent less than the available 5% error budget.

That protects against false winners, but it also costs power:

• At 2,000 visitors per day over 90 days, a real 10% lift on a 3% baseline reached about 81% power, with a median of roughly 86,000 visitors before a verdict.
• A 5% lift on a 1% baseline reached only about 7% power over the same horizon.

Being stricter than promised is not automatically better. It can make merchants wait longer than the disclosed risk requires. Future tuning is therefore evaluated on two axes at once: the false-positive budget must continue to hold, and time to decision should improve.

Parallel experiments

Apex supports multiple experiments on the same traffic. This is a designed-for case.

• Each experiment randomizes independently, so assignments from another experiment are distributed across both arms and normally cancel out of the comparison.
• Overlap can add variance because another treatment changes the environment for everyone.
• Material interactions are possible when two tests affect the same surface or mechanism. Those conflicts should be prevented with targeting and exclusion groups rather than by serializing the entire program.
• Sample-ratio checks and trust gating remain active for every experiment independently.

Parallel testing preserves learning velocity without assuming that only one experiment can exist per visitor. Apex attribution and analytics are designed to retain multiple experiment assignments.

Guardian and trust gates

Some measurements determine whether Apex should show a verdict at all.

• Guardian harm auto-pause: all 10 certified cells conformed, measuring 0.05% to 0.17% false pauses against a 1% budget.
• Sample-ratio mismatch: the check measured about 0.09% against its 0.1% nominal boundary and detected simulated 55/45 and 60/40 allocation corruption in every drill, with a median detection time of one day.
• Broken-variation tripwire: the baseline audit found up to 8.9% false flags under a 90/10 allocation. After adding an expected-conversions floor, the re-measured rate was at most 0.004%.
• Revenue harm check: all four cells violated the 1% budget, measuring 1.624% to 3.202% false pauses. It remains part of the revenue inference remediation.

If allocation or measurement integrity is broken, the correct response is to suppress the verdict, not decorate untrustworthy data with a confidence number.

The certification run

The certification grid was registered before the full run and executed through the production decision functions.

• More than 3,000,000 simulated experiments across 68 cells.
• Baseline rates from 0.5% to 10%.
• Traffic from 100 to 10,000 visitors per day.
• Two- and three-arm designs.
• 28- and 90-day horizons.
• Fixed seeds and deterministic derivation.
• Generated reports and machine-computed verdicts rather than hand-entered results.

Decision surface	Cells	Result	Measured outcome
Sequential winner verdict at 95%	24	24/24 conform	0.2%–3.9% vs 5%
Classical test under daily peeking	6	6/6 violate	27.1%–27.6% vs 5%
Guardian harm auto-pause	10	10/10 conform	0.05%–0.17% vs 1%
Revenue significance	4	4/4 violate	21.384%–27.284% vs 5%; gated
Revenue harm check	4	4/4 violate	1.624%–3.202% vs 1%
Guardian tripwire	6	Descriptive	at most 0.004% after remediation at 90/10
Sample-ratio check	3 budgeted plus 2 drills	Boundary-calibrated	about 0.09% vs 0.1%; all corruption drills detected
Power and time to decision	9	Descriptive	quantifies the cost of conservative decisions

Of the 51 cells with a pass/fail budget, 34 conformed, 14 violated, and 3 were marginal at the sample-ratio boundary. The remaining 17 cells measured descriptive properties such as power, detection speed, and tripwire noise.

What the certificate means

The certificate supports three claims:

1. The live sequential winner verdict kept its false-positive promise in the registered scenarios at the tested 95% configuration.
2. Guardian’s automated harm action was conservative in the registered scenarios.
3. Apex found and disclosed failures in other surfaces instead of treating the audit as marketing.

It does not prove that every possible traffic pattern or metric distribution is safe. It is a reproducible baseline and an acceptance harness: when statistical code changes, the affected scenarios must be measured again.

See also the audit of common Bayesian stopping rules.