A/B Test Sample Size and Early Stopping (2026)
By William Zhu & the InfiniSynapse Data Team · Published: 2026-08-22 · Last updated: 2026-08-24 · Last verified: 2026-08-24 · Next review: 2026-11-24 · Editorial standards · Corrections
Table of Contents
- TL;DR
- What Ab Test Sample Size Actually Locks
- A Stopping-Contract Framework
- How Teams Compare Horizons and Looks
- Tool Landscape for Honest Stopping
- Implementation Steps before the First Refresh
- Desk Sample: Illustrative Early-Stop Pack
- Selection Scorecard for Stopping Rules
- Failure Modes That Treat Refresh as Method
- Frequently Asked Questions
- Conclusion
TL;DR
We evaluate these patterns at the InfiniSynapse desk on sanitized composites; sample figures on this page are illustrative, not customer uplifts.
Direct answer: Ab test sample size is a stopping contract: a pre-registered horizon or a sequential rule with recorded looks. A daily refresh that stops at the first green day is not a sample-size method.
What you'll learn: why ab test sample size is a decision, not a calculator screenshot; a six-row stopping table; when early stop is valid; how CUPED does not replace n; an illustrative peek pack; and the breaks that inflate false wins.
Ab test sample size is not “how many rows we have today.” It is the rule for when you are allowed to decide. If neither a horizon nor a sequential plan exists, label the read exploratory.
What Ab Test Sample Size Actually Locks
Key Definition: Ab test sample size is the pre-registered count or information budget that tells a team when it may stop and decide, including any sequential looks, written in the same memo as assignment and guardrails. It is not a live badge that turns green.
Interim-analysis discipline at the FDA is the right analogy even when the change is a button: you plan the looks, or you do not get to claim a planned stop. Ab test sample size inherits that bar.
Ab test sample size sits under the parent method in A/B test analysis. This page is narrower: stopping, not the full assignment-first read. For that order, use experiment analysis. For the optional variance cut, use CUPED explained.
A calculator is not a contract
A power spreadsheet can be useful. Ab test sample size still needs the unit, the baseline, the minimum effect you care about, and the peeking rule. A number without those four is decoration. If stakeholders will look daily, the calculator that assumes one look is already wrong.
Published statistical series at Eurostat state the period and the revision policy. Ab test sample size should state the horizon and what happens if someone looks early.
Early stop is a decision
Stopping early can be valid under a sequential method with recorded look times. Ab test sample size that stops because “it is already significant” without a spending function is optimism. Write the look. Hide the primary until the horizon if your culture cannot resist the badge.
A Stopping-Contract Framework
Every pack that claims ab test sample size should fill this table before anyone debates lift.
| Contract row | What you lock | Typical source | Failure if skipped |
|---|---|---|---|
| Unit | User, account, or session | Assignment log | Power on the wrong grain |
| Baseline + MDE | Rate and smallest useful lift | Design note | Fantasy n |
| Horizon or budget | End date or information | Same note | Endless “one more day” |
| Peeking rule | None, or sequential | Same note | Daily theater |
| Look log | Who looked, when | Ops note | Forgotten peeks |
| Decision | Ship / hold / iterate owner | Memo | Chat as approval |
Ab test sample size quality is the filled contract. Cloud-control language in ISO/IEC 27017 is not a power textbook, but it is a reminder that access to a live board is a control. If everyone can refresh the primary, you have looks.
Peer-review culture at Science still expects the planned analysis. Ab test sample size that omits the plan is a story.
How Teams Compare Horizons and Looks
Teams argue tools. They should argue who is allowed to see the primary. Ab test sample size methods differ in what a look costs.
| Method family | Works when | Breaks when |
|---|---|---|
| Fixed horizon | No peeks, n pre-registered | Stakeholders refresh daily |
| Sequential / always-valid | Peeking is the culture | Look times unrecorded |
| CUPED-assisted n | Clean pre-period covariate | Cut used to shrink n after peek |
| Exploratory | Labeled as such | Sold as a ship-ready test |
Environmental monitoring at the EPA publishes sampling frequency. Ab test sample size should publish look frequency the same way: planned, or none.
Fixed horizon versus sequential peeking
Classic ab test sample size at a fixed horizon is honest if nobody looks. Most product orgs look. If you will look, pick a sequential procedure and record each look. Stopping at the first green day without a spending function is not conservative.
CUPED does not replace n
Variance reduction can change the n you planned. It does not let you ignore peeks. Ab test sample size that “rescues” a short test with CUPED after a disappointing look is two failures stacked. Keep the unadjusted plan. See CUPED explained for when the cut is even eligible.
Tool Landscape for Honest Stopping
You do not need a new warehouse to honor ab test sample size. You need the assignment log, a written horizon or sequential rule, and a memo that records looks. A dated export is valid if you freeze the file date and stop treating later appends as new looks without a rule.
A data agent can draft “n reached / not reached” and attach SQL. It cannot accept residual risk. AI for data analysis is the method parent; this page is the stopping clause.
Counters you can reopen
Minimum columns for ab test sample size: unit, variant, assigned_at, and a running eligible count. If exposure is not assignment, say whether n is ITT or exposed. Exploratory data analysis on the accumulation curve belongs here—before lift.
Knowledge-base sentences for MDE
“Minimum detectable effect” is a sentence: which primary, which window, which baseline. Bind it. Ab test sample size that lets each teammate pick a new MDE after a flat week will never end. InfiniSynapse binds a knowledge base to the data source you authorize; it does not ship a prebuilt power warehouse, and it does not ship the variant.
If you need the join grain behind the count, continue in analyze experiment results in SQL. If you need the artifact the owner signs, use experiment decision memo.
Implementation Steps before the First Refresh
Start with the stopping rule, not with the badge. Ab test sample size that starts at “is it significant yet” will stop early.
Freeze horizon or sequential looks
Write the end date or the information budget in one paragraph. Ab test sample size at session grain while you assign users will mis-count n. If a user can enter twice, stop and fix assignment. Record who may see the primary before the horizon.
Ask whether n is enough to decide
Ask for a decision memo: planned n, observed n, look log, primary, guardrails, CUPED on or off, recommended action. The agent writes the memo. A human decides. Open the SQL. Ab test sample size without the count query is a slide.
Guardrail metrics still veto a “n reached” win. Enough sample on a primary does not clear a refund miss.
Desk Sample: Illustrative Early-Stop Pack
The following numbers are an illustrative desk composite, not a customer result and not an uplift claim.
| Item | Desk composite (illustrative) |
|---|---|
| Planned n | 80,000 users, fixed 28-day horizon |
| Observed at day 11 | 31,200 users |
| Looks | Four unofficial dashboard refreshes |
| Primary at day 11 | +2.4 pp (illustrative) |
| Primary at day 28 | +0.6 pp (illustrative) |
| Guardrail | Refunds unchanged (illustrative) |
| Decision | Hold as exploratory; rerun with a look log |
Ab test sample size on this pack is useful because the early green day was not the end state. A memo that shipped at day 11 would have treated a refresh as a method.
Figure. Desk composite from this page: Planned 80k / 28 days; day-11 +2.4 pp on 31.2k; day-28 +0.6 pp. Published context: fda.gov; ec.europa.eu; iso.org. Not a customer experiment, SLA, or official benchmark.
| Evidence class | What you can cite | What you cannot claim |
|---|---|---|
| Desk composite on this page | Planned n, looks, later fade | Customer uplift or vendor bake-off |
| Published public sources above | Interim plans, periods, access, sampling | That those bodies ran this desk pack |
Desk composite: day-11 +2.4 pp faded to +0.6 pp → hold. Context: FDA interims, Eurostat periods, ISO 27017 access, Science planned analysis, EPA sampling frequency.
We ran this check on a sanitized composite at the InfiniSynapse desk on 2026-08-23. We asked ab test sample size on the authorized composite and reopened SQL before anyone briefed. The reject was first-green-day stops. The assignment table, the written primary, and both intervals had to be present or the pack was held. Figures stay illustrative. What you can copy is the assignment join and the on/off rule, not a lift.
Selection Scorecard for Stopping Rules
Score from 1 to 5. Ab test sample size that cannot show a look log should not win on a prettier badge.
| Criterion | What “5” looks like | Disqualifier |
|---|---|---|
| Planned n | Horizon or budget written | “We’ll know when we see it” |
| Peeking | Rule written or no looks | Daily “is it significant yet” |
| Look log | Times recorded | Forgotten refreshes |
| Grain | n on the randomized unit | Session n, user assignment |
| Audit | Memo + SQL downloadable | Chat-only winner text |
| Decision rights | Named human owner | Model “recommends ship” |
Ab test sample size scores well when a skeptical partner can replay the count. It scores poorly when the stack implies a prebuilt experiment warehouse you do not operate.
Failure Modes That Treat Refresh as Method
Write the break in the memo if it happened. Reviews go faster when unofficial looks are explicit.
First-green-day stops
Daily refreshes inflate false wins. Ab test sample size that stops at the first green day without a spending function is not conservative. Either pre-register looks or hide the primary until the horizon.
Changing MDE after a flat week
Lowering the effect you “care about” so that current n looks enough is thawing the design. Ab test sample size keeps the MDE from the note. Extend or rerun; do not rewrite the contract.
CUPED as an n rescue
Turning on an adjustment because the unadjusted n disappointed is two failures. Ab test sample size does not borrow confidence from a leaking covariate. Leave the cut off if you already peeked.
A fourth pattern is counting exposed-only rows as ITT n. Say which count you used. Do not mix them mid-test.
Before you open a workspace, check four things: planned n, peeking rule, look log, and unit grain. If those four are missing, a tool will still produce a confident interval.
Route the same diagnosis to the live guide that owns the next object. Each row is a single hop.
| Live guide | Open it when |
|---|---|
| chat with your data | you need a question on an authorized extract |
| self-service analytics | a non-analyst will refresh the board |
| explainable AI data analysis | the count SQL must be auditable |
Ask whether the sample is enough to decide
Upload a sanitized assignment extract, bind the planned n and peeking rule, and ask whether the sample is enough to decide. This check uses only sources you authorize.
Commercial association: You do not need the workspace to complete the educational diagnosis on this page.
Open InfiniSynapseHow this page is sourced. William Zhu is cofounder of InfiniSynapse (GitHub @allwefantasy); no personal LinkedIn is published. Desk experience: designing and reviewing production analysis packs—definition locks, read-only source binds, and downloadable
/tasksartifacts. Reviewed by analytics engineering · data platform · LLM security · editor. Editorial standards · corrections · publishing principles · Contact zhuhl@infinisynapse.com. Company Vision. COI: InfiniSynapse sells an AI-native Data Agent; the in-article banner is a commercial association. Fact-check: fda.gov · ec.europa.eu · ISO · science.org · epa.gov.
Frequently Asked Questions
Can I stop when the dashboard turns green?
Bottom line: Only if a sequential rule already paid for that look. Ab test sample size is not a refresh. Stopping at the first green day without a spending function is exploratory. Hide the primary until the horizon, or record every look and use a method that allows it.
Does CUPED let me run a smaller test?
Bottom line: Sometimes, if the covariate was in the plan and the pre-period is clean. Ab test sample size still needs a peeking rule. Do not cut n after you already looked. Keep the unadjusted plan in the memo.
What if we already peeked without a rule?
Bottom line: Label the read exploratory. Ab test sample size cannot be repaired by pretending the looks did not happen. Extend with a new rule, or rerun. Do not ship on the unofficial interval.
Who decides that n is enough?
Bottom line: A human with launch authority. Ab test sample size in an agent memo is a count and a rule. It is not a ship order. Guardrails can still veto. Write the owner.
Conclusion
Ab test sample size is a stopping contract: horizon or sequential looks, recorded, on the unit you randomized. A refresh is not a method. Do not rewrite MDE to match today’s n. Do not use CUPED as a rescue. Name the human who ships.
When the assignment table and the planned n are ready, ask whether the sample is enough on an authorized extract at https://app.infinisynapse.com/. Open the SQL, keep the look log, and reuse the same rule on the next test.