# Jev Benchmark Protocol for Data-Agent Decisions

**Version:** 1.0  
**Created:** 2026-09-23  
**Status:** Blank protocol; no InfiniSynapse run has been performed  
**Companion schema:** `jev-benchmark-results-template.csv`

## 1. Research question

For each declared decision family, does the tested Jev version meet the pre-registered accuracy, calibration, high-risk-error, latency, and total-cost requirements relative to:

1. deterministic rules;
2. the current production baseline; and
3. one strong-model baseline?

Do not create a composite “best model” score. Decide against requirements per metric.

## 2. Freeze before API access

Record and hash:

- protocol version;
- dataset commit and SHA-256;
- label guide;
- train/validation/test row IDs;
- question definitions and version;
- answer-option order and descriptions;
- model IDs and provider routes;
- timeout, attempts, concurrency, and client region;
- confidence threshold selected on validation data;
- minimum accuracy and Macro-F1;
- maximum Brier score and ECE;
- maximum high-risk-error rate;
- maximum p95 latency;
- maximum total cost per 1,000 completed decisions;
- go/no-go rule.

Never tune a threshold on the final test split.

## 3. Dataset

Use at least two real decision families. Recommended examples:

- route a request to one declared metric owner;
- classify tool risk as read-only, destructive, privileged, or exfiltration;
- judge whether evidence is sufficient for human review.

Sampling requirements:

- stratify by class and difficulty;
- preserve a later time slice for drift;
- include ambiguous, adversarial, and out-of-distribution rows;
- remove secrets and unnecessary personal data;
- document exclusions and licensing.

Two reviewers label each row independently. Adjudicate disagreements without seeing model outputs. Store `label_source`, reviewer IDs or pseudonyms, and adjudication status.

## 4. Systems

Test:

- deterministic rules, where applicable;
- current production baseline;
- Jev with an immutable version ID;
- one strong-model baseline;
- optional Jev-to-strong-model cascade.

Record native versus emulated probability interfaces. A Jev native distribution and an LLM-written confidence number are not interchangeable.

## 5. Execution

1. Randomize row order with a saved seed.
2. Warm each route with excluded requests and record cold-start behavior separately.
3. Run every system on the same test rows.
4. Use the same concurrency unless a provider restriction requires otherwise.
5. Save wall-clock latency for every attempt.
6. Count terminal failures and exhausted retries as wrong.
7. Save the resolved model ID, provider route, token usage, and billed cost.
8. Preserve the full declared probability distribution.
9. Do not omit timeouts, malformed responses, or refused requests.
10. Run cascades from pre-registered thresholds only.

## 6. Row-level fields

Use the companion CSV. `probabilities_json` must contain a JSON object with every declared option. Leave unavailable fields blank; never fill missing data with estimates.

Required identity fields:

- `run_id`, `row_id`, `decision_family`, `split`;
- `provider`, `requested_model`, `resolved_model`;
- `question_version`, `ground_truth`, `prediction`.

Required measurement fields:

- `correct`, `confidence`, `probabilities_json`;
- `latency_ms`, `input_tokens`, `output_tokens`, `billed_cost_usd`;
- `attempt_count`, `failure_class`;
- `escalated`, `fallback_model`, `human_review_seconds`;
- `high_risk_error`, `end_to_end_success`.

## 7. Metrics

### Accuracy and class performance

- Accuracy = correct attempts / all rows, with failures wrong.
- Macro-F1 = unweighted mean of class F1 values.
- Publish confusion matrices and per-class support.
- For paired model comparisons, use exact McNemar tests.
- Bootstrap confidence intervals by scenario or source unit when rows are correlated.

### Calibration

- Binary Brier = mean `(p - y)^2`.
- Multiclass Brier = mean sum over classes `(p_k - y_k)^2`.
- ECE = weighted absolute gap between bin confidence and bin accuracy.
- Declare bin count and boundary rule.
- Publish reliability-bin counts and a reliability curve.
- Publish selective accuracy and coverage at every evaluated threshold.

### Latency and reliability

- Report p50, p95, and maximum wall-clock latency.
- Report cold-start observations separately.
- Report success rate, timeout rate, malformed-response rate, and retries per completed decision.
- State whether network time was retained or derived away.

### Cost

Report:

- provider-billed cost where available;
- derived token cost only with dated, cited rates;
- Jev cost on every row;
- fallback cost on escalated rows;
- human-review cost as `seconds / 3600 × loaded_hourly_rate`;
- total cost per 1,000 completed decisions.

For a cascade:

`hybrid_cost(f) = jev_cost + f × fallback_cost + human_review_cost`

### Safety and completion

- High-risk-error rate = flagged high-risk wrong approvals / relevant rows.
- Escalation rate = escalated rows / all rows.
- Automation rate = non-escalated successful rows / all rows.
- End-to-end completion = rows whose full decision chain meets the acceptance rule.

## 8. Go/no-go rule

Write exact values before running. Example structure, not recommended thresholds:

```text
GO only if:
  accuracy >= [VALUE]
  macro_f1 >= [VALUE]
  brier <= [VALUE]
  ece <= [VALUE]
  high_risk_error_rate <= [VALUE]
  p95_latency_ms <= [VALUE]
  total_cost_per_1000 <= [VALUE]
  end_to_end_success >= [VALUE]
and every required metric has complete data.
```

If confidence intervals cross the required boundary, record `AMBIGUOUS`; do not relabel it a win.

## 9. Publication checklist

- [ ] Provider terms permit publication.
- [ ] Exact model versions and routes are named.
- [ ] Dataset license permits the published artifacts.
- [ ] Dataset hash, protocol, seed, and code are public.
- [ ] Row-level outputs or a legally shareable equivalent are public.
- [ ] Failures and retries remain in the denominator.
- [ ] Ground truth is distinguished from model agreement.
- [ ] Native and verbalized probabilities are not pooled.
- [ ] p50 and p95 include a stated network boundary.
- [ ] Costs identify provider, currency, rate date, and billing source.
- [ ] Thresholds came from validation, not test data.
- [ ] Losses and ambiguous results appear as prominently as wins.
- [ ] InfiniSynapse involvement and conflicts are disclosed.
- [ ] No claim implies current InfiniSynapse–Jev integration unless separately verified.

## 10. Evidence status

This protocol is an original, reusable evaluation asset. It contains no observations and is not evidence that Jev or any baseline meets a requirement. A completed run requires credentials, frozen artifacts, row-level outputs, analysis code, and a signed review.