Jev Benchmark: Accuracy, Calibration, Cost, Latency

By William Zhu & the InfiniSynapse Data Team · Published: 2026-09-23 · Last updated: 2026-09-23 · Last verified: 2026-09-23 · Next review: 2026-12-23 · Editorial standards · Corrections

Author credentials: William Zhu is Cofounder of InfiniSynapse, with a public engineering identity at GitHub @allwefantasy. He published the Apple 10-K tests cited below. This review combines those disclosed first-party observations with attributed independent results and a protocol for a future controlled run.

Table of Contents

• TL;DR
• What is publishable today
• Separate workflow evaluations from ground truth
• Public evidence snapshot
• Accuracy: report the task, not the brand
• Calibration: probabilities must earn trust
• Latency: publish p50, p95, and the route
• Cost: price the complete decision path
• Tail behavior and multi-decision correctness
• A reproducible benchmark design
• Decision rule for a data-agent pilot
• Evidence limits
• Frequently Asked Questions
• Conclusion

TL;DR

Direct answer: Public Jev benchmark evidence does not support one universal accuracy or speed number. On independent task-specific Jev benchmark tests reviewed here, version 1.13 ranges from 62.6% phishing accuracy to 95.4% held-out three-way email accuracy; another 791-decision head-to-head reports 78.8%–87.0% across three tasks, 0.33-second median latency, and task-dependent cost of $0.0136–$0.0400 per 1,000 decisions. The useful conclusion is narrower: a Jev benchmark can show a fast, inexpensive typed gate, but accuracy, calibration, and escalation thresholds must be measured on your labels.

This article is an evidence review and protocol, not a new InfiniSynapse experiment. It cites two earlier public tests run by William Zhu alongside independent sources, but the underlying row-level files are not bundled here and the founder results have not been independently reproduced. No current InfiniSynapse product integration is claimed. The downloadable benchmark protocol and row-level data template are ready for a future controlled run.

For third-party studies, we evaluated the published methods and artifacts as a desk review. For the founder rows, this page transcribes the reported public figures and limitations rather than presenting a rerun.

What is publishable today

Three Jev benchmark outputs are publishable: source-specific public results, a comparison of their methods, and a reproducible protocol that refuses to invent missing observations. A headline such as “Jev is 193× faster” is not publishable without its baseline, task shape, reasoning setting, geography, concurrency, and measurement boundary.

A responsible Jev benchmark publishes losses, failures, and ambiguity with the same prominence as favorable point estimates.

The independent JevBench methodology makes the core point explicit: a metric exists inside a run with a dataset, protocol, and date. Its evidence archive reports Jev benchmark accuracy spanning 62.6% to 95.4%. That range is not inconsistency; it is what happens when phishing, banking intents, tool risk, and email classification are different tasks.

The Stanford AI Index is methodological context because it keeps benchmark trends attached to named tests and documented limitations. It does not validate any Jev benchmark result on this page.

What remains unpublished here

InfiniSynapse did not run a new experiment for this article. The evidence table below includes a previously published Apple 10-K test by cofounder William Zhu, clearly labeled as founder-reported first-party evidence. The underlying row-level labels, traces, and invoices are not bundled with this article, so the figures are citable observations rather than an independently reproducible InfiniSynapse benchmark. They do not establish current product integration.

Separate workflow evaluations from ground truth

A Jev benchmark workflow evaluation asks whether a model reproduces decisions made by a reference model or completes a coded workflow. A ground-truth test asks whether its output matches independently defined labels. Both can be useful, but they answer different questions.

Vendor workflow evaluations can show the economics of batching several typed judgments and moving deterministic logic into code. They cannot establish real-world correctness when the “truth” is an average of other models’ probabilities. An independent Jev benchmark should report that as agreement, not accuracy.

The NIST AI Risk Management Framework treats measurement and evaluation as part of risk management, not as a one-number leaderboard. That is the standard applied here: define context, measure the relevant harm, and retain limitations.

Agreement is not correctness

The AY Automate 791-decision head-to-head illustrates the gap. It reports Jev agreement with GPT-5.6 Terra at 90.0%, 87.4%, and 93.8% on three tasks, while Jev accuracy against labels was 83.8%, 79.1%, and 87.0%. Agreement exceeded accuracy by roughly six to eight points.

For a data agent, the distinction is operational. A routing model can agree with a larger model and still choose the wrong metric owner. Build the answer key from reviewed business outcomes, not another model’s preferences.

Public evidence snapshot

The Jev benchmark table below keeps each number attached to its task. It does not average rows into a winner.

Figure. Public accuracy results from AY Automate: GPT-5.6 Terra is the comparator for routing and Claude Haiku 4.5 for injection detection. This is not InfiniSynapse testing.

The JevBench run archive labels reproducible artifacts separately from independent reproduction; at verification time it says no listed run had been independently reproduced. Public code raises confidence in inspectability, not automatic confidence in the result.

The founder’s six-system follow-up also reported that Jev accuracy stayed at 92.2% as its threshold moved from 5 to 9, while retaining all 25 gold documents. Treat that as a useful calibration observation for this corpus, not proof of calibration elsewhere. A separate 10-call availability spot check completed only 4 calls; its six failures make fallback testing necessary but are far too few to estimate service uptime.

Accuracy: report the task, not the brand

For a Jev benchmark, accuracy is the number correct divided by all attempted items, with failed requests counted as wrong. Report Macro-F1 as well when classes are imbalanced or mistakes across rare classes matter. For each model pair, use the same rows and a paired test such as exact McNemar rather than comparing overlapping confidence intervals by eye.

Every task-level Jev benchmark should retain the confusion matrix, class support, and failed calls needed to recompute that accuracy.

AY Automate found Jev statistically behind GPT-5.6 Terra on 77-way intent routing (Jev-only correct 7, Terra-only correct 19, p=0.029) and ahead of GPT-5.4 nano on prompt-injection detection (28 versus 3, p<0.001). That Jev benchmark did not show universal leadership. It showed task-dependent wins and losses.

High-risk errors deserve their own metric

In a Jev benchmark, a false approval for a destructive tool call is not equivalent to routing a support ticket to the second-best queue. Mark a high_risk_error column before the run, then report its rate and absolute count. Never bury it inside aggregate accuracy.

OWASP’s Top 10 for LLM Applications repository provides threat categories for the surrounding agent system. It is not a Jev test, but it helps define which routing errors deserve a separate high-consequence label.

Calibration: probabilities must earn trust

A Jev benchmark calibration analysis asks whether events assigned probability 0.8 occur about 80% of the time. Use multiclass Brier score over the full distribution and Expected Calibration Error with a declared binning rule. Plot a reliability curve and publish bin counts; ECE can look better or worse when bins change.

A calibration-focused Jev benchmark also reports coverage, because abstaining on difficult rows can improve selective accuracy while reducing automation.

WotAI’s 16-model calibration test reports Jev at 66.0% accuracy, ECE 0.121, and 34.7% of rows in the 0.35–0.65 “unsure” band. Haiku had the same accuracy, ECE 0.122, but only 2.7% unsure. Sonnet 5 had better ECE at 0.062 but a 1,674 ms median versus Jev’s 455 ms. These are properties of 150 prose passages, not a universal calibration ranking.

Confidence is not calibration

Choice confidence summarizes the returned distribution; it is not proof that a 0.9 threshold is safe. In AY Automate’s routing data, accuracy increased as the threshold rose, yet Jev still made 17 confident errors across the two intent tasks at 0.90 or above. Validate a threshold on held-out data and lock it before the test set.

Latency: publish p50, p95, and the route

A Jev benchmark must report wall-clock p50 and p95, client region, provider route, concurrency, timeout, retries, cold starts, and whether network time was subtracted. “Model latency” is not defensible when gateway and network time remain in the number.

Google’s Site Reliability Engineering book explains why tail behavior and service-level objectives matter beyond averages. Apply that operational lens to the complete decision path, not only the fastest successful call.

AY Automate measured 0.33 s p50 and 0.44 s p95 for Jev across 791 calls through OpenRouter, versus 0.67/0.89 s for Gemini 3.5 Flash-Lite and 1.17/2.54 s for GPT-5.6 Terra. Its laptop used four parallel requests per system. Those conditions belong beside the result.

Why published speedups vary from 1.7× to 100×

TrueStandard’s comparison-shape experiment measured one Jev decision at 477 ms server time versus 790 ms for Gemini 3.1 Flash Lite: 1.7×. It then compared six sequential judgments by a thinking model with one batched Jev call: 74.42 s versus 0.739 s, or 100.7×.

Both numbers can be true because the baseline changed from one fast classification to six sequential deliberative calls. A credible Jev benchmark names the unit: per request, per judgment, or end-to-end workflow.

Cost: price the complete decision path

A Jev benchmark should report billed cost from the provider when possible. If cost is derived, retain input tokens, output tokens, rates, currency, and rate-verification date. A cascade costs one Jev call for every item plus a fallback call for the escalated fraction:

A cost-focused Jev benchmark must include unsuccessful attempts and escalations rather than pricing only accepted answers.

hybrid_cost(f) = jev_cost + f × fallback_cost

AY Automate reports Jev at $0.0151, $0.0400, and $0.0136 per 1,000 decisions on its three tasks. GPT-5.6 Terra was $0.6089, $1.9569, and $0.6733. Those are OpenRouter billed costs for those prompts—not a first-party TypeSafe price guarantee and not a projection for your state length.

Include escalation and human review

At a 0.80 confidence gate, AY Automate sent 19.4% of 8-way rows and 23.0% of 77-way rows to Terra. The resulting cascade matched or slightly exceeded Terra-only point estimates at 25.7% and 27.9% of Terra-only cost. The threshold was evaluated on the same items, so it is an estimate that needs a separate validation set.

Human review is not free. Add review_seconds × loaded_hourly_rate to the cost model. A cheap model that escalates most cases can cost more end to end than a more expensive model with fewer reviews.

Tail behavior and multi-decision correctness

The LargitData multi-turn RAG routing study reports Jev 1.13.0 at 61.4% when all required decisions on a turn had to be correct, despite individual components such as route at 90.6% and mode at 91.6%. Its p95 was 818 ms versus Gemma 4 31B at 4,723 ms.

That result demonstrates why a data-agent Jev benchmark needs end-to-end completion in addition to per-question accuracy. Four 90% decisions do not produce a 90% workflow when every one must be right.

A reproducible benchmark design

Freeze tasks and labels

Define two or three real Jev benchmark decision families, such as metric-owner routing, evidence sufficiency, and tool-risk class. Sample by class and difficulty. Use two reviewers plus adjudication for disputed labels. Hash the frozen dataset before any model run.

The AWS Machine Learning Lens provides broader workload guidance for reproducibility, monitoring, security, reliability, and cost. Use it to review the benchmark environment; do not treat it as evidence for Jev performance.

Use honest baselines

Include deterministic rules, the current production model, a strong model, and Jev. Give every model the same evidence and answer space where possible. Record when an LLM receives a JSON schema while Jev receives native typed questions; these are different interfaces and must remain labeled.

Pre-register thresholds and exclusions

Choose the confidence gate on a validation split. Pre-register timeout, maximum attempts, failed-call treatment, minimum accuracy, maximum high-risk error rate, and the go/no-go rule. Do not tune on the test set and then describe the threshold as prospective.

Capture every request

The downloadable row template contains identifiers, labels, predictions, full-distribution JSON, latency, token usage, billed cost, attempt count, failure class, escalation, human time, and high-risk error flags. The protocol defines splits, metrics, calculations, and a publication checklist.

Decision rule for a data-agent pilot

Adopt Jev only if the Jev benchmark meets the pre-registered minimum accuracy and high-risk ceiling, improves total cost or latency after fallback, and produces a stable escalation rate on a later time slice. A model may win p50 and still fail the release because p95, calibration, or high-risk errors miss the bar.

Place the result inside the data infrastructure runtime and attach it to a decision job. Use Jev vs LLM to choose the appropriate baseline and Jev for AI agents to define the routing boundary. The proposed Jev API integration guide handles safe execution, while data ops handles resume and audit. This page owns evidence, not deployment.

Evidence limits

The reviewed public Jev benchmark runs use different datasets, providers, dates, model identifiers, prompts, concurrency, and metrics. Some use ground truth; others use model agreement or bespoke labels. Public artifacts have not necessarily been rerun by a second party. Pricing and model aliases can change. Training-data contamination cannot be ruled out for public datasets.

No weighted average across these studies is defensible. The Jev benchmark search task is complete only when each number remains attached to its method and the reader can download a protocol for a local test.

Frequently Asked Questions

Is there one authoritative Jev accuracy number?

Bottom line: No. Public Jev benchmark task results reviewed here range from 62.6% to 95.4%. Quote the dataset, sample, model version, protocol, and date with every number.

Is Jev always 193 times faster?

Bottom line: No. Independent comparisons reviewed here range from about 1.7× for one fast-model decision to 100.7× when six deliberative calls are collapsed into one batched request.

Does good confidence mean Jev is calibrated?

Bottom line: Not by itself. Measure Brier score, ECE, reliability bins, selective accuracy, and coverage on held-out labels from your task.

Has InfiniSynapse run this benchmark?

Bottom line: William Zhu published two first-party Apple 10-K tests that are cited and labeled above. InfiniSynapse has not run the downloadable protocol as a new benchmark for this article, and the founder results have not been independently reproduced.

Can I publish results produced with the download?

Bottom line: Only after checking current provider terms, fixing model versions and rates, publishing failures and exclusions, and making the dataset and row-level outputs available where licensing permits.

Conclusion

The honest Jev benchmark conclusion is conditional. Jev can be fast, cheap, and useful for confidence-aware typed gates, but it is not uniformly the most accurate or best calibrated system. Ground-truth accuracy, calibration, p95 latency, complete cost, escalation, and high-risk errors must be measured together. Until InfiniSynapse runs the attached protocol, this page remains an evidence review—not a product result or integration claim.