Experiment Analysis after the Design Is Frozen (2026)
By William Zhu & the InfiniSynapse Data Team · Published: 2026-08-22 · Last updated: 2026-08-31 · Last verified: 2026-08-31 · Next review: 2026-11-30 · About · Privacy policy · Editorial standards · Corrections
Table of Contents
- TL;DR
- What Experiment Analysis Means after Freeze
- A Read-Order Framework for Frozen Tests
- How Teams Compare Post-Design Reads
- Tool Landscape for Assignment-First Reads
- Implementation Steps after the Design Note
- Accuracy and Experience Record: Illustrative Assignment-First Pack
- Evidence Boundaries and Independent Validation
- How to Cite This Page
- Selection Scorecard for Post-Design Reads
- Failure Modes That Ignore the Freeze
- Frequently Asked Questions
- Conclusion
TL;DR
We evaluate these patterns at the InfiniSynapse desk on sanitized composites; sample figures on this page are illustrative, not customer uplifts.
Direct answer: Experiment analysis after freeze is a read of the assignment table first—unit, variant, time, eligibility—then outcomes, guardrails, and a memo a human signs. Lift is the last number, not the first slide.
What you'll learn: why experiment analysis starts with assignment integrity; a six-row read-order table; how ITT and exposed-only differ; how to bind metric notes; an illustrative checkout pack; and the breaks that reopen a “frozen” design.
Download evidence: desk log · assignment CSV · verification script · source check · reproduction protocol. This package is first-party and illustrative—not customer, production, randomized-trial, benchmark, or third-party evidence.
Experiment analysis is not a dashboard refresh. It is a contract replay. If the design note is missing, label the pack exploratory and stop treating the interval as a launch order.
What Experiment Analysis Means after Freeze
Key Definition: Experiment analysis is the post-design read of a frozen assignment log against named outcomes and guardrails, producing a ship / hold / iterate memo a reviewer can audit. The freeze locks unit, primary, window, and peeking rule; it does not lock the ship decision.
U.S. Census Bureau materials (retrieved 2026-09-04) provide official-statistics context. Sampling frames are an analogy here, not experimental validation.
For direct statistical guidance, see the ASA Statement on Statistical Significance and P-Values and NIST/SEMATECH e-Handbook of Statistical Methods (retrieved 2026-09-04). Neither reviewed this page or run.
Experiment analysis sits next to the hub method in A/B test analysis. This page is narrower: what you do after the note is signed, not how you pick CUPED or write the first hypothesis.
Assignment is the object, lift is the comment
Experiment analysis that opens on conversion will shop for a winner. Open the assignment table: one unit, one variant, an assigned_at that precedes every outcome you will count. If a user appears in both arms, stop. If timestamps are missing, stop. Exploratory data analysis on balance and sample-ratio mismatch belongs here, before any lift sentence.
CMS materials (retrieved 2026-09-04) concern U.S. health programs. Code integrity is an analogy, not evidence for this experiment.
Freeze is a protocol, not a mood
FDA materials and the specific Adaptive Designs guidance (retrieved 2026-09-04) concern regulated clinical trials. Planning principles are informative, not direct product-test rules.
If the next object is variance reduction rather than assignment order, continue in CUPED explained. If the next object is a kill metric, use guardrail metrics.
A Read-Order Framework for Frozen Tests
Every experiment analysis read should fill this table before anyone debates lift.
| Read order | What you lock | Typical source | Failure if skipped |
|---|---|---|---|
| 1. Unit | User, account, or session | Assignment log | Cookie splits one person |
| 2. Variant | One row per unit | Same log | Dual-exposed users |
| 3. Time | assigned_at before outcomes | Same log | Pre-period counted as lift |
| 4. Eligibility | Who was allowed in | Flag or join | Ineligible “wins” |
| 5. Primary + guardrails | One primary, named kills | Outcome + metric note | Metric shopping |
| 6. Decision owner | Human who ships | Memo | Chat as approval |
Experiment analysis quality is the filled order, not the novelty of the estimator. If refunds are a guardrail and the extract has no refund grain, write “not measured.” Hiding the gap is worse than a hold.
data.gov (retrieved 2026-09-04) is a government data catalog. Dataset vintage is a publication analogy only.
How Teams Compare Post-Design Reads
Teams argue tools. After freeze they should argue inclusion. Experiment analysis methods differ in which rows they keep.
| Inclusion family | Works when | Breaks when |
|---|---|---|
| Intent-to-treat | Assignment is the policy | Exposure is sold as ITT |
| Exposed-only | Exposure is logged cleanly | Exposure is inferred from clicks |
| Per-protocol | Compliance is rare and named | Non-compliance is dropped quietly |
| Slice-first | Pre-registered segments | Segments invented after lift |
UK Office for National Statistics materials (retrieved 2026-09-04) provide official-statistics context, not review of this method.
Intent-to-treat versus exposed-only
Classic experiment analysis is intent-to-treat: you analyze the unit you randomized. Exposed-only is a different question—did people who saw the UI convert?—and it is easy to bias. If you use it, say so, and keep the ITT line in the memo. A data agent can compute both; it cannot choose the estimand for you.
Slices that reopen the design
A slice that was not in the note is a new test. Experiment analysis may report it as exploratory. It may not promote it to the primary because the original interval looked small. That promotion is how frozen designs thaw overnight.
Tool Landscape for Assignment-First Reads
You do not need a new warehouse to finish experiment analysis. You need the assignment log, the outcome grain, and a metric note bound to those sources. A dated CSV is valid if it includes assignment time.
AI for data analysis can draft the memo. It is not NLP2SQL theater and it is not ChatBI. The human still names the primary and signs the last line.
Assignment logs you can reopen
Minimum columns: unit_id, variant, assigned_at, eligibility. Experiment analysis that lacks those four is a story. If randomization lived in another system, keep that system’s IDs in the extract. Data governance at experiment grain means one definition of “assigned,” not three Slack threads.
Metric notes bound to sources
“Conversion” is a sentence. Bind it. Experiment analysis that lets each teammate redefine conversion will produce two memos. InfiniSynapse binds a knowledge base to the data source you authorize; it does not ship a prebuilt metric warehouse, and it does not write the ship decision back into production.
Implementation Steps after the Design Note
Start from the note, not from “is it significant.” Experiment analysis that starts at the badge will shop for a metric.
Replay assignment before outcomes
Confirm uniqueness, variant balance, and that outcomes sit after assignment. Experiment analysis on a join that uses first-event time instead of assigned_at will manufacture lift. Record sample-ratio mismatch. A 48/52 split on a 50/50 design is a diagnosis, not a footnote.
If you will peek, the note must already name the sequential rule. Daily refreshes are not a method. For horizon math, see A/B test sample size.
Bind guardrails and ask for the memo
List two or three kill metrics before you compute the primary. Typical experiment analysis kills: refund rate, p95 latency, support contacts, unsubscribe. If a join is missing, write “not measured.” Then ask for a decision memo: primary, interval, guardrails, CUPED on or off, balance, recommended action, and the SQL.
The agent writes the memo. A human decides ship, hold, or iterate. Open the query. Experiment analysis without attached SQL is a slide. If you need the downloadable artifact shape, continue in experiment decision memo. If you need the join itself, use analyze experiment results in SQL.
Accuracy and Experience Record: Illustrative Assignment-First Pack
The following numbers are an illustrative desk composite, not a customer result or uplift claim. Run ID: EA-FREEZE-20260823. Run date: 2026-08-23. Operator: InfiniSynapse Data Team. Objects inspected: assignment count and split, primary, refund guardrail, SRM evidence gap, five aggregates, and two held items.
| Item | Desk composite (illustrative) |
|---|---|
| Window | 28 days after freeze |
| Units | 64,200 users; 50.4% / 49.6% |
| SRM note | Mild 50.4/49.6; assignment job checked |
| Primary | Checkout completion |
| Observed delta | +1.1 percentage points (illustrative) |
| Guardrail | Refund rate +0.3 percentage points (illustrative) |
| Decision | Hold; refund join was late and still adverse |
Experiment analysis on this pack is useful because assignment was read first. A memo that opened on +1.1 pp would have looked like a ship. The late refund join is the finding.
Figure. Desk composite from this page: 64,200 users; +1.1 pp checkout; +0.3 pp refund; mild SRM noted. Published context: census.gov; cms.gov; fda.gov. Not a customer experiment, SLA, or official benchmark.
| Evidence class | What you can cite | What you cannot claim |
|---|---|---|
| Desk composite on this page | Grain, order, inspectable artifacts | Customer uplift or vendor bake-off |
| Published public sources above | Frames, protocols, published extracts | That those agencies ran this desk pack |
Desk composite: +1.1 pp primary, +0.3 pp refund → hold. Context: Census frames, CMS integrity, FDA protocol freeze, data.gov vintages, ONS exclusions.
The operator withheld an SRM conclusion and held shipping for the late adverse refund join. The desk log records those limitations. The CSV exposes five illustrative aggregates and two held items.
Evidence Boundaries and Independent Validation
This is not customer, production, randomized-trial, peer-reviewed, benchmark, representative, or causal evidence. Assignment/outcome rows, expected allocation, intervals, variances, SQL, stopping rule, look history, and SRM test are unavailable.
The 64,200 users are not a disclosed sampling frame. The 50.4%/49.6% split and 1.1-/0.3-point deltas cannot be independently recomputed. Calling the split “mild SRM” is not a statistical conclusion without the planned allocation and test result.
The source check distinguishes statistical references from agency analogies. The open protocol specifies an external test. As of 2026-08-31, no qualifying independent report, statistical peer review, customer validation, or media investigation exists.
The output checker confirms displayed labels and values only. It does not establish SRM, randomization integrity, significance, ITT validity, causation, or commercial impact.
Experiment analysis reads assignment. Experiment analysis checks uniqueness. Experiment analysis verifies timing. Experiment analysis records allocation. Experiment analysis tests SRM. Experiment analysis preserves ITT denominators. Experiment analysis labels exploratory slices. Experiment analysis reports guardrail gaps. Experiment analysis attaches SQL artifacts. Experiment analysis leaves humans accountable.
How to Cite This Page
Page: Zhu, W., & InfiniSynapse Data Team. (2026). Experiment analysis after the design is frozen. InfiniSynapse. https://infinisynapse.com/en/blog/experiment-analysis
Run: InfiniSynapse Data Team. (2026). Desk log EA-FREEZE-20260823 (illustrative assignment composite). https://infinisynapse.com/blog-media/experiment-analysis/downloads/desk-log-EA-FREEZE-20260823.md
Neither is an independent audit, customer experiment, randomized trial, peer review, benchmark, or proof of lift. Cite unavailable rows, withheld SRM conclusion, late refund join, held decision, and first-party limitations.
Selection Scorecard for Post-Design Reads
Score from 1 to 5. Experiment analysis that cannot show assignment SQL should not win on a prettier badge.
| Criterion | What “5” looks like | Disqualifier |
|---|---|---|
| Assignment first | Unique unit, timestamps, SRM | Lift-first dashboards |
| Freeze honesty | Note unchanged after peek | Primary rewritten mid-test |
| Inclusion named | ITT or exposed, written | Silent row drops |
| Guardrails | Pre-registered and joined | “Fine” with no column |
| Audit | Memo + SQL downloadable | Chat-only winner text |
| Decision rights | Named human owner | Model “recommends ship” as policy |
Experiment analysis scores well when a skeptical partner can replay the read. It scores poorly when the stack implies a prebuilt experiment warehouse you do not operate.
Failure Modes That Ignore the Freeze
Write the break in the memo if it happened. Reviews go faster when invalidations are explicit.
Lift before assignment
Opening on conversion hides dual-exposed users and bad clocks. Experiment analysis that skips uniqueness will discover the bug in the launch email. Fix the log, then compute.
Thawing the primary after a peek
Changing the primary because the original looked flat is a new test. Experiment analysis may attach the exploratory chart. It may not treat the new metric as the frozen contract.
Guardrail silence after freeze
A conversion win that raises refunds is not a product win. Experiment analysis that cannot join a kill metric must say so. “Not in the extract” is a hold reason, not a pass.
A fourth pattern is treating the agent paragraph as the ship order. The memo is a draft. The owner’s name is the decision.
Before you open a workspace, check four things: unit uniqueness, assignment timestamps, a written primary, and at least one guardrail join. If those four are missing, a tool will still produce a confident interval.
Route the same diagnosis to the live guide that owns the next object. Each row is a single hop.
| Live guide | Open it when |
|---|---|
| explainable AI data analysis | the plan and SQL must be auditable |
| ecommerce analytics | orders and refunds sit in different sources |
| data knowledge base | metric sentences need a bound note |
Ask for the memo from the assignment table
Upload a sanitized assignment-and-outcome extract, bind the metric note, and ask for a memo that starts with assignment integrity. This check uses only sources you authorize.
Commercial association: You do not need the workspace to complete the educational diagnosis on this page.
Open InfiniSynapseHow this page is sourced. William Zhu is cofounder of InfiniSynapse (GitHub @allwefantasy); no personal LinkedIn, statistics credential, experiment-platform affiliation, or independent reviewer role is claimed. His profile establishes authorship, not statistical qualification. Desk decisions are recorded in run EA-FREEZE-20260823. Reviewed internally by analytics engineering · data platform · LLM security · editor. Editorial standards · corrections · publishing principles · Company Vision. COI: InfiniSynapse sells an AI-native Data Agent. ASA, NIST, FDA, Census, CMS, data.gov, and ONS did not validate this run.
Frequently Asked Questions
Do I read lift first if the design is already frozen?
Bottom line: No. Experiment analysis after freeze still starts with assignment. The freeze locks the contract; it does not prove the log is clean. Check uniqueness, timestamps, eligibility, and SRM before any lift sentence. If those fail, the interval is not a launch order.
Can I switch from ITT to exposed-only after seeing the number?
Bottom line: Not as the primary. Experiment analysis may report exposed-only as a secondary, labeled exploratory, if exposure is logged. Promoting it after a disappointing ITT is thawing the design. Keep both lines if you compute both, and say which one the owner is deciding on.
Is a CSV export enough for a frozen test?
Bottom line: Yes, if it includes unit, variant, assignment time, and outcomes. Experiment analysis on a file is still analysis. Freeze the file date. Bind metric definitions. Do not treat a missing refund column as a zero refund rate. If randomization lived elsewhere, keep those IDs.
Who signs the memo if an agent drafted it?
Bottom line: A human with launch authority. Experiment analysis can produce the intervals and the SQL. It cannot accept residual risk. Write the owner on the document. If marketing wants a public claim, raise the evidence bar before copy ships.
Can readers recompute the split and both deltas?
Bottom line: No. Assignment/outcome rows and inferential outputs are unavailable. The CSV makes five aggregates and two held items inspectable, not independently reproducible.
Has an independent statistician reproduced this run?
Bottom line: No qualifying external report is published as of 2026-08-31. The protocol defines the data, pre-registration, diagnostics, and review required.
Conclusion
Experiment analysis after freeze is assignment first, then outcomes, then a memo. The freeze is a protocol: unit, primary, window, peeking rule. It is not a license to skip SRM or hide a missing guardrail. Hold when the log is dirty. Label slices that were not in the note. Name the human who ships.
When the assignment table and the metric note are ready, ask for that memo on an authorized extract at https://app.infinisynapse.com/. Open the SQL, keep the file, and reuse the same freeze on the next test.