A/B Test Sample Size and Early Stopping (2026)
By William Zhu & the InfiniSynapse Data Team · Published: 2026-08-22 · Last updated: 2026-08-31 · Last verified: 2026-08-31 · Next review: 2026-11-30 · About · Privacy policy · Editorial standards · Corrections
Table of Contents
- TL;DR
- What Ab Test Sample Size Actually Locks
- A Stopping-Contract Framework
- How Teams Compare Horizons and Looks
- Tool Landscape for Honest Stopping
- Implementation Steps before the First Refresh
- Accuracy and Experience Record: Illustrative Early-Stop Pack
- Evidence Boundaries and Independent Validation
- How to Cite This Page
- Selection Scorecard for Stopping Rules
- Failure Modes That Treat Refresh as Method
- Frequently Asked Questions
- Conclusion
TL;DR
We evaluate these patterns at the InfiniSynapse desk on sanitized composites; sample figures on this page are illustrative, not customer uplifts.
Direct answer: Ab test sample size is a stopping contract: a pre-registered horizon or a sequential rule with recorded looks. A daily refresh that stops at the first green day is not a sample-size method.
What you'll learn: why ab test sample size is a decision, not a calculator screenshot; a six-row stopping table; when early stop is valid; how CUPED does not replace n; an illustrative peek pack; and the breaks that inflate false wins.
Download evidence: desk log · stopping CSV · verification script · source check · reproduction protocol. This package is first-party and illustrative—not customer, production, randomized-trial, benchmark, or third-party evidence.
Ab test sample size is not “how many rows we have today.” It is the rule for when you are allowed to decide. If neither a horizon nor a sequential plan exists, label the read exploratory.
What Ab Test Sample Size Actually Locks
Key Definition: Ab test sample size is the pre-registered count or information budget that tells a team when it may stop and decide, including any sequential looks, written in the same memo as assignment and guardrails. It is not a live badge that turns green.
FDA materials (retrieved 2026-09-04) are broad regulatory context. The specific FDA Adaptive Designs guidance addresses clinical trials; its planning principles are informative, not direct rules for product experiments.
For direct statistical reference, use the NIST/SEMATECH e-Handbook of Statistical Methods and ASA Statement on Statistical Significance and P-Values (retrieved 2026-09-04). Neither reviewed this page or run.
Ab test sample size sits under the parent method in A/B test analysis. This page is narrower: stopping, not the full assignment-first read. For that order, use experiment analysis. For the optional variance cut, use CUPED explained.
A calculator is not a contract
A power spreadsheet can be useful. Ab test sample size still needs the unit, the baseline, the minimum effect you care about, and the peeking rule. A number without those four is decoration. If stakeholders will look daily, the calculator that assumes one look is already wrong.
Eurostat (retrieved 2026-09-04) is an official statistics publisher. Its role here is a publication-period analogy, not experimental-design authority.
Early stop is a decision
Stopping early can be valid under a sequential method with recorded look times. Ab test sample size that stops because “it is already significant” without a spending function is optimism. Write the look. Hide the primary until the horizon if your culture cannot resist the badge.
A Stopping-Contract Framework
Every pack that claims ab test sample size should fill this table before anyone debates lift.
| Contract row | What you lock | Typical source | Failure if skipped |
|---|---|---|---|
| Unit | User, account, or session | Assignment log | Power on the wrong grain |
| Baseline + MDE | Rate and smallest useful lift | Design note | Fantasy n |
| Horizon or budget | End date or information | Same note | Endless “one more day” |
| Peeking rule | None, or sequential | Same note | Daily theater |
| Look log | Who looked, when | Ops note | Forgotten peeks |
| Decision | Ship / hold / iterate owner | Memo | Chat as approval |
Ab test sample size quality is the filled contract. ISO/IEC 27017 (retrieved 2026-09-04) concerns cloud security controls, not power calculations or stopping rules.
Science (retrieved 2026-09-04) is a journal publisher. It does not independently review this article or substitute for a cited study.
How Teams Compare Horizons and Looks
Teams argue tools. They should argue who is allowed to see the primary. Ab test sample size methods differ in what a look costs.
| Method family | Works when | Breaks when |
|---|---|---|
| Fixed horizon | No peeks, n pre-registered | Stakeholders refresh daily |
| Sequential / always-valid | Peeking is the culture | Look times unrecorded |
| CUPED-assisted n | Clean pre-period covariate | Cut used to shrink n after peek |
| Exploratory | Labeled as such | Sold as a ship-ready test |
EPA materials (retrieved 2026-09-04) concern environmental protection. Sampling frequency is an analogy only.
Fixed horizon versus sequential peeking
Classic ab test sample size at a fixed horizon is honest if nobody looks. Most product orgs look. If you will look, pick a sequential procedure and record each look. Stopping at the first green day without a spending function is not conservative.
CUPED does not replace n
Variance reduction can change the n you planned. It does not let you ignore peeks. Ab test sample size that “rescues” a short test with CUPED after a disappointing look is two failures stacked. Keep the unadjusted plan. See CUPED explained for when the cut is even eligible.
Tool Landscape for Honest Stopping
You do not need a new warehouse to honor ab test sample size. You need the assignment log, a written horizon or sequential rule, and a memo that records looks. A dated export is valid if you freeze the file date and stop treating later appends as new looks without a rule.
A data agent can draft “n reached / not reached” and attach SQL. It cannot accept residual risk. AI for data analysis is the method parent; this page is the stopping clause.
Counters you can reopen
Minimum columns for ab test sample size: unit, variant, assigned_at, and a running eligible count. If exposure is not assignment, say whether n is ITT or exposed. Exploratory data analysis on the accumulation curve belongs here—before lift.
Knowledge-base sentences for MDE
“Minimum detectable effect” is a sentence: which primary, which window, which baseline. Bind it. Ab test sample size that lets each teammate pick a new MDE after a flat week will never end. InfiniSynapse binds a knowledge base to the data source you authorize; it does not ship a prebuilt power warehouse, and it does not ship the variant.
If you need the join grain behind the count, continue in analyze experiment results in SQL. If you need the artifact the owner signs, use experiment decision memo.
Implementation Steps before the First Refresh
Start with the stopping rule, not with the badge. Ab test sample size that starts at “is it significant yet” will stop early.
Freeze horizon or sequential looks
Write the end date or the information budget in one paragraph. Ab test sample size at session grain while you assign users will mis-count n. If a user can enter twice, stop and fix assignment. Record who may see the primary before the horizon.
Ask whether n is enough to decide
Ask for a decision memo: planned n, observed n, look log, primary, guardrails, CUPED on or off, recommended action. The agent writes the memo. A human decides. Open the SQL. Ab test sample size without the count query is a slide.
Guardrail metrics still veto a “n reached” win. Enough sample on a primary does not clear a refund miss.
Accuracy and Experience Record: Illustrative Early-Stop Pack
The following numbers are an illustrative desk composite, not a customer result or uplift claim. Run ID: ABSS-PEEK-20260823. Run date: 2026-08-23. Operator: InfiniSynapse Data Team. Objects inspected: planned and observed n, four unofficial looks, day-11/day-28 deltas, refund label, and two held decisions.
| Item | Desk composite (illustrative) |
|---|---|
| Planned n | 80,000 users, fixed 28-day horizon |
| Observed at day 11 | 31,200 users |
| Looks | Four unofficial dashboard refreshes |
| Primary at day 11 | +2.4 pp (illustrative) |
| Primary at day 28 | +0.6 pp (illustrative) |
| Guardrail | Refunds unchanged (illustrative) |
| Decision | Hold as exploratory; rerun with a look log |
Ab test sample size on this pack is useful because the early green day was not the end state. A memo that shipped at day 11 would have treated a refresh as a method.
Figure. Desk composite from this page: Planned 80k / 28 days; day-11 +2.4 pp on 31.2k; day-28 +0.6 pp. Published context: fda.gov; ec.europa.eu; iso.org. Not a customer experiment, SLA, or official benchmark.
| Evidence class | What you can cite | What you cannot claim |
|---|---|---|
| Desk composite on this page | Planned n, looks, later fade | Customer uplift or vendor bake-off |
| Published public sources above | Interim plans, periods, access, sampling | That those bodies ran this desk pack |
Desk composite: day-11 +2.4 pp faded to +0.6 pp → hold. Context: FDA interims, Eurostat periods, ISO 27017 access, Science planned analysis, EPA sampling frequency.
The operator rejected the first-green-day stop. The desk log records the two held decisions. The CSV exposes six aggregates.
Evidence Boundaries and Independent Validation
This is not customer, production, randomized-trial, peer-reviewed, benchmark, representative, or causal evidence. Assignment/outcome rows, baseline, MDE, alpha, power, variances, intervals, code, exact look timestamps, and guardrail definition are unavailable.
The 80,000 planned and 31,200 observed users are not disclosed sampling frames. The four looks and 2.4-/0.6-point deltas cannot be independently recomputed. No power calculation, sequential boundary, p-value, confidence interval, or multiplicity correction is published.
The source check distinguishes statistical references from analogies. The open protocol specifies an external test. As of 2026-08-31, no qualifying independent report, statistical peer review, customer validation, or media investigation exists.
The output checker confirms displayed labels and values only. It does not establish sample adequacy, randomization integrity, significance, early-stopping validity, causation, or commercial impact.
Ab test sample size fixes eligibility. Ab test sample size records baseline. Ab test sample size declares effects. Ab test sample size states power. Ab test sample size locks error rates. Ab test sample size defines units. Ab test sample size pre-registers looks. Ab test sample size records looks. Ab test sample size reports missingness. Ab test sample size preserves vetoes. Ab test sample size assigns accountability.
How to Cite This Page
Page: Zhu, W., & InfiniSynapse Data Team. (2026). A/B test sample size and early stopping. InfiniSynapse. https://infinisynapse.com/en/blog/ab-test-sample-size
Run: InfiniSynapse Data Team. (2026). Desk log ABSS-PEEK-20260823 (illustrative stopping composite). https://infinisynapse.com/blog-media/ab-test-sample-size/downloads/desk-log-ABSS-PEEK-20260823.md
Neither is an independent audit, customer experiment, randomized trial, peer review, calculator validation, benchmark, or proof of lift. Cite unavailable rows, missing design inputs, held decisions, and first-party limitations.
Selection Scorecard for Stopping Rules
Score from 1 to 5. Ab test sample size that cannot show a look log should not win on a prettier badge.
| Criterion | What “5” looks like | Disqualifier |
|---|---|---|
| Planned n | Horizon or budget written | “We’ll know when we see it” |
| Peeking | Rule written or no looks | Daily “is it significant yet” |
| Look log | Times recorded | Forgotten refreshes |
| Grain | n on the randomized unit | Session n, user assignment |
| Audit | Memo + SQL downloadable | Chat-only winner text |
| Decision rights | Named human owner | Model “recommends ship” |
Ab test sample size scores well when a skeptical partner can replay the count. It scores poorly when the stack implies a prebuilt experiment warehouse you do not operate.
Failure Modes That Treat Refresh as Method
Write the break in the memo if it happened. Reviews go faster when unofficial looks are explicit.
First-green-day stops
Daily refreshes inflate false wins. Ab test sample size that stops at the first green day without a spending function is not conservative. Either pre-register looks or hide the primary until the horizon.
Changing MDE after a flat week
Lowering the effect you “care about” so that current n looks enough is thawing the design. Ab test sample size keeps the MDE from the note. Extend or rerun; do not rewrite the contract.
CUPED as an n rescue
Turning on an adjustment because the unadjusted n disappointed is two failures. Ab test sample size does not borrow confidence from a leaking covariate. Leave the cut off if you already peeked.
A fourth pattern is counting exposed-only rows as ITT n. Say which count you used. Do not mix them mid-test.
Before you open a workspace, check four things: planned n, peeking rule, look log, and unit grain. If those four are missing, a tool will still produce a confident interval.
Route the same diagnosis to the live guide that owns the next object. Each row is a single hop.
| Live guide | Open it when |
|---|---|
| chat with your data | you need a question on an authorized extract |
| self-service analytics | a non-analyst will refresh the board |
| explainable AI data analysis | the count SQL must be auditable |
Ask whether the sample is enough to decide
Upload a sanitized assignment extract, bind the planned n and peeking rule, and ask whether the sample is enough to decide. This check uses only sources you authorize.
Commercial association: You do not need the workspace to complete the educational diagnosis on this page.
Open InfiniSynapseHow this page is sourced. William Zhu is cofounder of InfiniSynapse (GitHub @allwefantasy); no personal LinkedIn, statistics credential, experiment-platform affiliation, or independent reviewer role is claimed. His profile establishes authorship, not statistical qualification. Desk decisions are recorded in run ABSS-PEEK-20260823. Reviewed internally by analytics engineering · data platform · LLM security · editor. Editorial standards · corrections · publishing principles · Company Vision. COI: InfiniSynapse sells an AI-native Data Agent. NIST, ASA, FDA, Eurostat, ISO, Science, and EPA did not validate this run.
Frequently Asked Questions
Can I stop when the dashboard turns green?
Bottom line: Only if a sequential rule already paid for that look. Ab test sample size is not a refresh. Stopping at the first green day without a spending function is exploratory. Hide the primary until the horizon, or record every look and use a method that allows it.
Does CUPED let me run a smaller test?
Bottom line: Sometimes, if the covariate was in the plan and the pre-period is clean. Ab test sample size still needs a peeking rule. Do not cut n after you already looked. Keep the unadjusted plan in the memo.
What if we already peeked without a rule?
Bottom line: Label the read exploratory. Ab test sample size cannot be repaired by pretending the looks did not happen. Extend with a new rule, or rerun. Do not ship on the unofficial interval.
Who decides that n is enough?
Bottom line: A human with launch authority. Ab test sample size in an agent memo is a count and a rule. It is not a ship order. Guardrails can still veto. Write the owner.
Can readers recompute the day-11 and day-28 deltas?
Bottom line: No. Assignment/outcome rows and design inputs are unavailable. The CSV makes six aggregates and two held decisions inspectable, not reproducible.
Has an independent statistician reproduced this run?
Bottom line: No qualifying external report is published as of 2026-08-31. The protocol defines the data, pre-registration, calculations, and review required.
Conclusion
Ab test sample size is a stopping contract: horizon or sequential looks, recorded, on the unit you randomized. A refresh is not a method. Do not rewrite MDE to match today’s n. Do not use CUPED as a rescue. Name the human who ships.
When the assignment table and the planned n are ready, ask whether the sample is enough on an authorized extract at https://app.infinisynapse.com/. Open the SQL, keep the look log, and reuse the same rule on the next test.