Experiment Decision Memo You Can Download (2026)
By William Zhu & the InfiniSynapse Data Team · Published: 2026-08-22 · Last updated: 2026-08-24 · Last verified: 2026-08-24 · Next review: 2026-11-24 · Editorial standards · Corrections
Table of Contents
- TL;DR
- What an Experiment Decision Memo Must Contain
- A Memo-Field Framework You Can Replay
- How Teams Compare Artifacts and Screenshots
- Tool Landscape for Downloadable Memos
- Implementation Steps from Tables to a File
- Desk Sample: Illustrative Memo Pack
- Selection Scorecard for Memo Stacks
- Failure Modes That Leave Only a Chat
- Frequently Asked Questions
- Conclusion
TL;DR
We evaluate these patterns at the InfiniSynapse desk on sanitized composites; sample figures on this page are illustrative, not customer uplifts.
Direct answer: An experiment decision memo is the downloadable artifact: assignment integrity, primary, interval, guardrails, CUPED on or off, attached SQL, and a human ship / hold / iterate line. The p-value is one row, not the product.
What you'll learn: why the experiment decision memo is the object you keep; a field table; how chat paragraphs fail audit; how to bind metric notes; an illustrative hold pack; and the breaks that leave only a screenshot.
An experiment decision memo is not a Slack verdict. It is a file a skeptic can reopen next quarter. If the SQL is missing, you have a slide.
What an Experiment Decision Memo Must Contain
Key Definition: An experiment decision memo is a signed, downloadable record of a frozen test—unit, primary, guardrails, peeking, CUPED state, queries, and a recommended action—that a human still accepts or rejects. It is not a chat paragraph that says “ship.”
Indicator publications at the OECD keep the definition next to the number. An experiment decision memo needs the same pairing: every metric is a sentence bound to a source, not an adjective.
An experiment decision memo sits under the parent method in A/B test analysis. This page is narrower: the artifact, not the assignment-first order. For that order, use experiment analysis. For the optional cut, use CUPED explained.
The p-value is a line
A significance badge answers a narrow question. An experiment decision memo answers a broader one: did we measure the unit we randomized, did we harm a kill, did we peek, and who ships? The badge can be in the table. It cannot be the title.
Statistical releases at Eurostat keep vintage and revision notes. An experiment decision memo should keep file date, window, and whether later events were appended.
Downloadable means inspectable
If you cannot download the experiment decision memo with the SQL that produced it, you cannot audit it. Task workspaces that produce Markdown, PDF, or data files are the right shape. A disappearing vendor UI is not. Standards language at the IEC is a reminder that a record has a lifetime after the meeting.
A Memo-Field Framework You Can Replay
Every experiment decision memo should fill this table before anyone treats the last sentence as a launch.
| Field | What you lock | Typical source | Failure if skipped |
|---|---|---|---|
| Assignment | Unit, variant, time, SRM | Assignment log | Dual-exposed users |
| Primary | One metric, one window | Outcome + note | Metric shopping |
| Guardrails | Named kills + availability | Adjacent tables | Silent harm |
| Peeking | Horizon or look log | Design note | First-green-day |
| CUPED | Covariate or off | Pre-period | Hidden toggle |
| Query | SQL or equivalent | Task trail | Slide-only |
| Owner | Human action | Signature | Chat as approval |
An experiment decision memo is the filled table, not a narrative that hides the gaps. Preprint methods on arXiv can inform an estimator; they do not replace the field list.
Research-program documentation at the NSF is a useful analogy for a packet you can reopen: question, method, data, limitation. An experiment decision memo is that packet at experiment grain.
How Teams Compare Artifacts and Screenshots
Teams argue tools. They should argue what remains after the meeting. An experiment decision memo beats a screenshot because the query travels with the claim.
| Artifact family | Works when | Breaks when |
|---|---|---|
| Downloadable memo + SQL | Skeptic can replay | File is a screenshot of a chat |
| Vendor UI export | Export includes query + vintage | Project ends, UI disappears |
| Notebook | Pinned revision, named owner | Local-only, no metric note |
| Chat verdict | Never, as the record | Used as the ship order |
Chat is a draft
An agent can draft the experiment decision memo. That draft is not the decision. AI for data analysis is a professional read, not ChatBI. The human still names the action.
Screenshots age badly
A badge without the join will not survive a finance review. An experiment decision memo that attaches the query and the metric note will. Explainable AI data analysis is the habit for opening the trail.
Tool Landscape for Downloadable Memos
You do not need a new warehouse to produce an experiment decision memo. You need assignment, outcomes, a bound metric note, and a workspace that keeps the file. A dated CSV is valid if the memo stamps the file date.
A data agent can write the memo and keep the SQL in the task trail. InfiniSynapse does that on sources you authorize. It does not auto-write the ship decision into production, and it does not ship a prebuilt experiment warehouse.
Tables plus a knowledge-base note
“Conversion” and “refund” are sentences. Bind them. An experiment decision memo that lets each teammate redefine conversion will fork. Data knowledge base is the hop when the note is the missing object. Data governance at experiment grain is the same bind.
Files you can keep
Markdown or PDF plus the query is enough. An experiment decision memo can live in a task workspace next to charts and extracts. If you need the join itself, continue in analyze experiment results in SQL. If you need kill lines inside the file, use guardrail metrics.
Implementation Steps from Tables to a File
Start from the design note, not from a winner adjective. An experiment decision memo that starts at “ship” will omit the gaps.
Freeze fields, then ask for the file
Confirm unit, primary, kills, peeking, and CUPED state. An experiment decision memo cannot invent a missing refund column. Write “not measured.” Ask for the downloadable memo with SQL. Open the joins. If assignment time was replaced by first-event time, reject the file.
Reuse the same field list on the next test so conversion does not drift. If the extract is a file, stamp the vintage in the header. If the extract is a live database, write the query time. Those two stamps are how a later reviewer knows which rows you saw.
Separate the paragraph from the signature
The last line is a human action: ship, hold, or iterate. An experiment decision memo does not ship itself. Write the owner. If marketing wants a public claim, raise the evidence bar before copy ships.
For horizon math behind the file, see A/B test sample size. For durable reuse of the same note, organizational analysis memory is the hop when the next test should not redefine conversion.
Desk Sample: Illustrative Memo Pack
The following numbers are an illustrative desk composite, not a customer result and not an uplift claim.
| Item | Desk composite (illustrative) |
|---|---|
| File | Markdown memo + SQL, dated extract |
| Units | 86,000 users; 50.1% / 49.9% |
| Primary | Checkout completion +1.8 pp (illustrative) |
| Guardrail | Refund rate +0.4 pp (illustrative) |
| CUPED | Prior 28-day orders; both lines shown |
| Peeking | No unofficial looks recorded |
| Decision | Hold; owner named; iterate retry copy |
An experiment decision memo on this pack is useful because the hold and the SQL are in the same file. A chat that said “ship the +1.8” would have dropped the kill. Keep the file next to the extract. If someone later asks why the launch waited, the refund line and the owner name are already there.
Figure. Desk composite from this page: 86,000 users; +1.8 pp primary, +0.4 pp refund; CUPED both lines in the memo. Published context: oecd.org; ec.europa.eu; iec.ch. Not a customer experiment, SLA, or official benchmark.
| Evidence class | What you can cite | What you cannot claim |
|---|---|---|
| Desk composite on this page | Fields, file shape, hold | Customer uplift or vendor bake-off |
| Published public sources above | Definitions, vintages, records | That those bodies ran this desk pack |
Desk composite: +1.8 pp primary, +0.4 pp refund → hold in the downloaded file. Context: OECD indicators, Eurostat vintage, IEC records, arXiv methods, NSF packets.
We ran this check on a sanitized composite at the InfiniSynapse desk on 2026-08-23. We bound the note, then asked one experiment decision memo question. We kept the memo only after the assignment table, the written primary, and both intervals were visible. We rejected chat as the record. Figures stay illustrative. What you can copy is the assignment join and the on/off rule, not a lift.
Selection Scorecard for Memo Stacks
Score from 1 to 5. An experiment decision memo that cannot be downloaded with SQL should not win on a prettier badge.
| Criterion | What “5” looks like | Disqualifier |
|---|---|---|
| Completeness | All contract fields present | Primary-only paragraph |
| Attachments | SQL + vintage + note | Screenshot of a chat |
| Guardrails | Availability written | Missing = fine |
| CUPED honesty | Named or off | Hidden toggle |
| Replay | Skeptic can rerun | Local-only notebook |
| Decision rights | Named human owner | Model “recommends ship” |
An experiment decision memo scores well when finance can reopen it next quarter. It scores poorly when the stack implies a prebuilt experiment warehouse you do not operate.
Failure Modes That Leave Only a Chat
Write the break in the memo if it happened. Reviews go faster when missing fields are explicit.
Chat as the record
A winner paragraph in a thread will disappear. An experiment decision memo that was never downloaded is not an artifact. Keep the file in the task workspace.
SQL omitted
A claim without a join is a slide. An experiment decision memo must attach the query or an equivalent inspectable plan. If the agent drafted SQL, read the window. If you wrote it, still attach it.
Owner left blank
The last line without a name is not a decision. An experiment decision memo needs a human who can accept residual risk. The agent does not ship.
A fourth pattern is attaching a screenshot of the badge and calling it the memo. The badge is a line. The file is the product.
Before you open a workspace, check four things: field list, metric note, assignment timestamps, and a named owner slot. If those four are missing, a tool will still produce a confident paragraph.
Route the same diagnosis to the live guide that owns the next object. Each row is a single hop.
| Live guide | Open it when |
|---|---|
| AI data report generator | you need a downloadable analysis pack |
| dashboard | someone wants tiles instead of a memo |
| self-service data analysis for business | a non-analyst will request the file |
Download the decision memo from the last run
Upload a sanitized assignment-and-outcome extract, bind the metric note, and ask to download the decision memo with SQL from the last run. This check uses only sources you authorize.
Commercial association: You do not need the workspace to complete the educational diagnosis on this page.
Open InfiniSynapseHow this page is sourced. William Zhu is cofounder of InfiniSynapse (GitHub @allwefantasy); no personal LinkedIn is published. Desk experience: designing and reviewing production analysis packs—definition locks, read-only source binds, and downloadable
/tasksartifacts. Reviewed by analytics engineering · data platform · LLM security · editor. Editorial standards · corrections · publishing principles · Contact zhuhl@infinisynapse.com. Company Vision. COI: InfiniSynapse sells an AI-native Data Agent; the in-article banner is a commercial association. Fact-check: oecd.org · ec.europa.eu · iec.ch · arxiv.org · nsf.gov.
Frequently Asked Questions
Is a p-value enough if the memo is short?
Bottom line: No. An experiment decision memo is the artifact; the p-value is a line. You still need assignment integrity, guardrails, peeking, CUPED state, SQL, and a named owner. A short memo that omits a kill is a slide.
Can I keep the memo as a chat transcript?
Bottom line: Not as the record. An experiment decision memo must be downloadable with the query. Chat can be the draft. Promote the file. Freeze the extract date so later appends do not silently rewrite history.
What if a guardrail column is missing?
Bottom line: Write “not measured” in the experiment decision memo. Do not infer zero. A hold is allowed. Bind the definition when the column arrives and rerun. The file should show the gap, not hide it.
Who signs if an agent wrote the draft?
Bottom line: A human with launch authority. An experiment decision memo can be drafted by an agent. It cannot accept residual risk. Write the owner. If marketing wants a public claim, raise the evidence bar before copy ships.
Conclusion
An experiment decision memo is the file you keep: fields, SQL, vintage, and a human line. The p-value is not the product. Hold when a kill moves or a join is missing. Keep CUPED named or off. Do not let a chat paragraph ship the variant.
When the assignment table and the metric note are ready, download that memo from an authorized run at https://app.infinisynapse.com/. Open the SQL, keep the file, and reuse the same fields on the next test.