Jev Benchmark Results: Accuracy and Latency

By William Zhu & the InfiniSynapse Data Team · Published: 2026-09-23 · Last updated: 2026-09-26 · Last verified: 2026-09-26 · Next review: 2026-12-23 · Editorial standards · Corrections

Author credentials: William Zhu is Cofounder of InfiniSynapse, with a public engineering identity at GitHub @allwefantasy. He published the Apple 10-K tests cited below. This review combines those disclosed first-party observations with attributed independent results and a protocol for a future controlled run.

Table of Contents

• TL;DR
• What is publishable today
• Separate workflow evaluations from ground truth
• Public evidence snapshot
• Accuracy: report the task, not the brand
• Calibration: probabilities must earn trust
• Latency: publish p50, p95, and the route
• Cost: price the complete decision path
• Tail behavior and multi-decision correctness
• A reproducible benchmark design
• Decision rule for a data-agent pilot
• Evidence limits
• Frequently Asked Questions
• Conclusion

TL;DR

Public Jev benchmark results reviewed here span 62.6% to 95.4% by task. A Jev benchmark workflow evaluation asks whether a model reproduces a reference model's decisions or completes a coded workflow. A ground-truth test asks whether the output matches independently defined labels. On the independent tests below, version 1.13 also shows 78.8%–87.0% accuracy across three labeled tasks, a 0.33-second median latency, and $0.0136–$0.0400 per 1,000 decisions. Quote the task, sample, version, and date with every number.

This article is an evidence review and protocol, not a new InfiniSynapse experiment. Third-party rows are a desk review of published methods. The Apple 10-K rows are founder-reported and have not been independently reproduced. No current product integration is claimed. The downloadable benchmark protocol and row-level template are ready for a later controlled run.

What is publishable today

Three Jev benchmark outputs are publishable: source-specific public results, a comparison of their methods, and a reproducible protocol that refuses to invent missing observations. A headline such as “Jev is 193× faster” is not publishable without its baseline, task shape, reasoning setting, geography, concurrency, and measurement boundary.

A responsible Jev benchmark publishes losses with the same prominence as favorable point estimates. The JevBench methodology ties every metric to a dataset, protocol, and date. The Stanford AI Index is context for that habit. It does not validate any result on this page.

What remains unpublished here. InfiniSynapse did not run a new experiment for this article. The table includes a previously published Apple 10-K test by cofounder William Zhu, labeled as founder-reported evidence. Row-level labels, traces, and invoices are not bundled here.

Separate workflow evaluations from ground truth

Report model-to-model agreement as agreement. A ground-truth Jev benchmark needs labels that were not taken from another model's probabilities. The NIST AI Risk Management Framework treats that measurement as risk management, not a one-number leaderboard.

Those questions are bounded. LangChain's harness guide documents three types: choice, score, and noul. MindStudio's hands-on notes on Jev 1.13.0 show why one prompt is not a Jev benchmark: a negated refund request dropped refund probability to 3%, and a forced three-way choice with no "other" still returned "sales" at 0.31. Eight synthetic cases do not set a threshold.

Agreement is not correctness.

The AY Automate 791-decision head-to-head reports Jev agreement with GPT-5.6 Terra at 90.0%, 87.4%, and 93.8%, while label accuracy was 83.8%, 79.1%, and 87.0%. Build the answer key from reviewed business outcomes. A route can match a larger model and still pick the wrong metric owner.

Public evidence snapshot

The Jev benchmark table below keeps each number attached to its task. It does not average rows into a winner.

Figure. Public accuracy results from AY Automate: GPT-5.6 Terra is the comparator for routing and Claude Haiku 4.5 for injection detection. This is not InfiniSynapse testing.

The JevBench run archive labels reproducible artifacts separately from independent reproduction; at verification time it says no listed run had been independently reproduced. Public code raises confidence in inspectability, not automatic confidence in the result.

Other published Jev benchmark figures use different denominators, so they stay out of the table above.

The Open-Jev benchmark index records Jev 1.13.0 at 200/231 (86.58%) on the public JevBench subset and 81/111 on Hard, with 303 of 534 tasks unavailable. JF100 is 232/300 across three rotations of 100 items, not 300 independent problems.

Layer3 Labs and DataCamp both surface TypeSafe's dashboard: 67.8% aggregate accuracy against 74.1% for an unnamed comparator, with task scores from 61.7% to 76.0%. DataCamp's summary places GPT-5.6 Terra at 67.9% and GPT-5.6 Sol at 74.1% on that 4-workflow set. That is a vendor dashboard, not a labeled comparison from the table.

Wavect's review of the Laya typed-decisions card reports 76.6% hard-label accuracy for a specialist fine-tuned on that training split, against a separately published 72.7% for Jev. The Laya author did not call Jev in that run. On the same card, soft accuracy and ECE favor the published Jev figures (0.580 versus 0.471; ECE 0.144 versus 0.213).

LangWatch's comparison adds 95% intervals on 15 tasks against open models under 1B, and states that Jev's training data is undisclosed, so contamination cannot be checked. This page does not copy that ranking.

The founder’s six-system follow-up kept Jev at 92.2% as its threshold moved from 5 to 9, with all 25 gold documents retained. That is a calibration note for this corpus only.

Accuracy: report the task, not the brand

For a Jev benchmark, accuracy is correct items divided by all attempts, with failed requests counted as wrong. Report Macro-F1 when classes are imbalanced. Compare models on the same rows with a paired test such as exact McNemar, and keep the confusion matrix.

AY Automate found Jev statistically behind GPT-5.6 Terra on 77-way intent routing (Jev-only correct 7, Terra-only correct 19, p=0.029) and ahead of GPT-5.4 nano on prompt-injection detection (28 versus 3, p<0.001). That Jev benchmark did not show universal leadership. It showed task-dependent wins and losses.

High-risk errors deserve their own metric

In a Jev benchmark, a false approval for a destructive tool call is not equivalent to routing a support ticket to the second-best queue. Mark a high_risk_error column before the run, then report its rate and absolute count. Never bury it inside aggregate accuracy.

OWASP’s Top 10 for LLM Applications repository provides threat categories for the surrounding agent system. It is not a Jev test, but it helps define which routing errors deserve a separate high-consequence label.

Calibration: probabilities must earn trust

A Jev benchmark calibration check asks whether events assigned probability 0.8 occur about 80% of the time. Publish multiclass Brier score, Expected Calibration Error with a declared binning rule, bin counts, and coverage. Abstaining can raise selective accuracy while automating fewer rows.

WotAI’s 16-model calibration test reports Jev at 66.0% accuracy, ECE 0.121, and 34.7% of rows in the 0.35–0.65 “unsure” band. Haiku had the same accuracy, ECE 0.122, but only 2.7% unsure. Sonnet 5 had better ECE at 0.062 but a 1,674 ms median versus Jev’s 455 ms. These are properties of 150 prose passages, not a universal calibration ranking.

Confidence is not calibration

Choice confidence summarizes the returned distribution; it is not proof that a 0.9 threshold is safe. In AY Automate’s routing data, accuracy increased as the threshold rose, yet Jev still made 17 confident errors across the two intent tasks at 0.90 or above. Validate a threshold on held-out data and lock it before the test set.

Option order belongs in the same Jev benchmark record. Regolo reports that reversing categorical options moved probabilities from 0.84–0.89 to 0.93–0.96 on identical inputs, and that reference documents placed first yielded 12 of 16 correct classifications versus 16 of 16 when placed last. Freeze option order, or rotate it and publish both.

Latency: publish p50, p95, and the route

A Jev benchmark reports wall-clock p50 and p95 with client region, route, concurrency, timeouts, retries, and cold starts. Google’s Site Reliability Engineering book is why the tail of the full decision path matters more than the fastest successful call.

AY Automate measured 0.33 s p50 and 0.44 s p95 for Jev across 791 calls through OpenRouter, versus 0.67/0.89 s for Gemini 3.5 Flash-Lite and 1.17/2.54 s for GPT-5.6 Terra. Its laptop used four parallel requests per system. Those conditions belong beside the result. Open-Jev's separate JevBench timing diagnostic, one observation per public task, reports Jev 1.13.0 at p50 291.3 ms and p95 353.7 ms. That timer is a separate Jev benchmark diagnostic, not this OpenRouter median.

Why published speedups vary from 1.7× to 100×

TrueStandard’s comparison-shape experiment measured one Jev decision at 477 ms server time versus 790 ms for Gemini 3.1 Flash Lite: 1.7×. It then compared six sequential judgments by a thinking model with one batched Jev call: 74.42 s versus 0.739 s, or 100.7×.

Both numbers can be true because the baseline changed from one fast classification to six sequential deliberative calls. A credible Jev benchmark names the unit: per request, per judgment, or end-to-end workflow.

Cost: price the complete decision path

A Jev benchmark should use billed cost when the provider shows it. If cost is derived, keep tokens, rates, currency, and the rate date. Price failed calls and escalations, not only accepted answers. A cascade is one Jev call per item plus a fallback on the escalated fraction:

hybrid_cost(f) = jev_cost + f × fallback_cost

AY Automate reports Jev at $0.0151, $0.0400, and $0.0136 per 1,000 decisions on its three tasks. GPT-5.6 Terra was $0.6089, $1.9569, and $0.6733. Those are OpenRouter billed costs for those prompts—not a first-party TypeSafe price guarantee and not a projection for your state length.

Include escalation and human review. At a 0.80 confidence gate, AY Automate sent 19.4% of 8-way rows and 23.0% of 77-way rows to Terra. The resulting cascade matched or slightly exceeded Terra-only point estimates at 25.7% and 27.9% of Terra-only cost. The threshold was evaluated on the same items, so it is an estimate that needs a separate validation set.

Human review is part of Jev benchmark cost. Add review_seconds × loaded_hourly_rate to the model. A cheap model that escalates most cases can cost more end to end than a more expensive model with fewer reviews.

Tail behavior and multi-decision correctness

The LargitData multi-turn RAG routing study reports Jev 1.13.0 at 61.4% when all required decisions on a turn had to be correct, despite individual components such as route at 90.6% and mode at 91.6%. Its p95 was 818 ms versus Gemma 4 31B at 4,723 ms.

That result demonstrates why a data-agent Jev benchmark needs end-to-end completion in addition to per-question accuracy. Four 90% decisions do not produce a 90% workflow when every one must be right.

A reproducible benchmark design

Freeze tasks and labels

Define two or three real Jev benchmark families, such as metric-owner routing, evidence sufficiency, and tool-risk class. Sample by class, adjudicate disputes, and hash the frozen set before any run. The AWS Machine Learning Lens is environment guidance, not evidence of Jev performance.

Use honest baselines

An honest Jev benchmark includes rules, the production model, a strong model, and Jev on the same evidence and answer space. Label it when an LLM gets a JSON schema and Jev gets native typed questions.

Pre-register thresholds and exclusions

A prospective Jev benchmark chooses the confidence gate on a validation split. Pre-register timeout, retries, failed-call treatment, minimum accuracy, the high-risk ceiling, and the go/no-go rule before opening the test set.

Capture every request

The downloadable row template contains identifiers, labels, predictions, full-distribution JSON, latency, token usage, billed cost, attempt count, failure class, escalation, human time, and high-risk error flags. The protocol defines the Jev benchmark splits, metrics, calculations, and publication checklist.

Decision rule for a data-agent pilot

Adopt Jev only if the Jev benchmark meets the pre-registered minimum accuracy and high-risk ceiling, improves total cost or latency after fallback, and produces a stable escalation rate on a later time slice. A model may win p50 and still fail the release because p95, calibration, or high-risk errors miss the bar.

Place the result inside the data infrastructure runtime and attach it to a decision job. Use Jev LLM to choose the appropriate baseline and the Jev agent routing boundary to decide where the route may sit. The proposed Jev API integration guide handles safe execution, while data ops handles resume and audit. This page owns the Jev benchmark evidence, not deployment.

Evidence limits

These Jev benchmark runs do not share a dataset, provider, date, or metric, and contamination cannot be ruled out. Do not average them. TypeSafe's Jev 1.13 notes, as cited by Wavect and Layer3 Labs, flag unreliable arithmetic, date comparison, and adversarial inputs. The model returns a typed decision, not an explanation or an image judgment. What Jev limits exclude keeps those jobs out of an analysis workflow. Keep those misses out of the accuracy denominator, and keep each published number attached to its method.

Frequently Asked Questions

Is Jev always 193 times faster?

Bottom line: No. Independent Jev benchmark comparisons reviewed here range from about 1.7× for one fast-model decision to 100.7× when six deliberative calls are collapsed into one batched request.

Does good confidence mean Jev is calibrated?

Bottom line: Not by itself. A Jev benchmark measures Brier score, ECE, reliability bins, selective accuracy, and coverage on held-out labels from your task.

Has InfiniSynapse run this benchmark?

Bottom line: William Zhu published two first-party Apple 10-K tests that are cited and labeled above. InfiniSynapse has not run this Jev benchmark protocol as a new test, and the founder results have not been independently reproduced.

Can I publish results produced with the download?

Bottom line: Only after checking current provider terms, fixing model versions and rates, publishing failures and exclusions, and making the dataset and row-level outputs available where licensing permits.

What accuracy does the Jev benchmark report?

Bottom line: No single score. Independent task results reviewed here run from 62.6% to 95.4%. TypeSafe's dashboard, reported separately above, is a 67.8% aggregate against an unnamed 74.1% comparator.

How does Jev compare with open checkpoints?

Bottom line: Only on a matched task. A fine-tuned Laya specialist's published 76.6% hard-label accuracy is not a zero-shot ranking, and Open-Jev's public JevBench subset is 200/231 for Jev 1.13.0. Record the checkpoint and whether Jev was called in that run.

What should a Jev benchmark exclude?

Bottom line: Arithmetic, date comparison, text generation, and unlabeled vendor aggregates. Score typed decisions against frozen labels, and keep high-risk errors in their own column.

Conclusion

The honest Jev benchmark conclusion is conditional. Jev can be a fast, cheap typed gate on some tasks and lose on others. Until InfiniSynapse runs the attached protocol, this page is an evidence review, not a product result.