benchmarks / methodology & evidence
Jev benchmarks: methodology and evidence first
Reject detached general leaderboard hype. We focus on real-world engineering metrics: categorical precision, tail latency, per-thousand-decision cost, and calibrated confidence curves.
Quick Answer: What is the Jev benchmark methodology?
Jev benchmarking prioritizes reproducible evidence over promotional rankings. Rather than testing generic trivia, our methodology evaluates task-specific enterprise decisions across four pillars: Macro F1 (balancing minority classes), P95 tail latency, per-decision token expense, and Expected Calibration Error (ECE) for reliable fallback thresholding.
Task Fixtures
Three Standardized Task Fixtures
Jev Support Ticket Routing Benchmark
A transparent fixture measuring categorical routing accuracy, tail latency, cost per 1k decisions, and human fallback trade-offs on labeled customer queries.
Jev Spam Detection & Moderation Benchmark
A task-level evaluation isolating false positives, stealth marketing evasion, inference latency, and quarantine queue trade-offs.
Jev Prompt Injection & Safety Defense Benchmark
An adversarial benchmark evaluating defense against instruction overrides, tool hijack attempts, context poisoning, and benign instruction preservation.
community boards / third-party evidence
Community & third-party Jev benchmarks
A search for “Jev benchmark” does not land on one scoreboard. Beyond our three fixtures above, a parallel ecosystem of community-run and third-party boards measures decision models — each board is a decision model benchmark with its own methodology, version and winner. These are all the boards we cite on this site: what each one actually measures, who maintains it, and the caveats its own maintainers disclose.
| Benchmark | Maintainer | Method & version | Key numbers | Relation to official numbers |
|---|---|---|---|---|
| ImageJevBench v0.1.3 | Community board for image-input decision models | Held-out comparison of systems that decide from images: 684 scored items (228 public + 456 sealed, plus 93 further items retired unscored), 49 systems ranked; Cost or Calibration under the gate scores 0. | Imajev-4B 76.39 · Jev-Omni 73.10 · NeoHorse Jev 4B 71.94 · Visual-Jev 4B 69.81 · JPT-4B 69.55 | The image-input axis, where hosted Jev did not compete at capture time. The card itself discloses that its 333 fresh sealed items are synthetic — easier for frontier API models. |
| JevBench v1.4.2 (independent counter-ledger) | Independent, MIT-licensed | Four-axis blended score across 93 systems, with a separate fresh-sealed split. Board versions have shipped with moving scores — the version-drift warning on our alternatives hub applies here too. | decider-4b v2 #1 · Jev 1.13.0 #2 (63.29) · JevK5 #3 (62.04) · GLiNER2.5-multi 63.1 · GLiNER2.5-small 62.1 | Different bench, different winner: on Fastino's own board GLiNER2.5-Decide leads, here decider-4b v2 does. The fresh-sealed split scored Jev 37% and JevK5 33% — a set its own evaluator calls unusually hard. |
| Jev Decision Index (community) | multimodalart, Hugging Face community | 70 open reproductions, 120K decisions per model across 43 benchmarks; suite v0.2.1, updated 2026-09-28. | Jev 57.9 #1 (524 ms hosted median) vs pplx-decider-v1 56.4 #4 (101 ms); Perplexity's v1.1 card reports 61.56 — the lead has flipped | Rankings move monthly and the v1.1 numbers are vendor-reported. Quote the current card, never a snapshot — including ours. |
| Liquid Decision Index v0.2.1 | Liquid AI (vendor) | Vendor-run board for the d1 family; public split, vendor-reported. | d1-3B 48.57 — first under 10B, on par with a ~35B model; d1-omni-600M 15.95; 7-task text mean 82.9 (d1-3B) | An in-house family board: meaningful inside the d1 lineup, not a cross-vendor ranking. An independent re-run on a fanless laptop measured roughly 3× the vendor's latency. |
| fast-decisions (Fastino) | Fastino (vendor) | 17 public subsets × 300 held-out examples = 5,100 cases, all English, exact-match scoring; footnoted on screen as “Fastino's internal benchmark”. | GLiNER2.5-Decide 60.2 · Decide-1B 59.6 · JevK5 57.6 · GLiNER2.5-multi 56.7 · Laya 46.6 | Fastino benchmarked JevK5 — the Apache-2.0 reproduction — not TypeSafe's hosted Jev, and GLiNER has never been run on JevBench: no third-party head-to-head of the two exists. |
| benchmarkheaven.com | Third-party live leaderboard | Live, unaudited snapshot; capability and composite scores change between checks. | Jev 1.13.0 capability 64.7 / composite 63.3; Laya #41 at 49.9 / 30.3, ~$0.0029 per 1,000 decisions (as captured in our Ollaya guide) | Weather, not climate: useful for orientation, not citable as a certified benchmark — the numbers drift. |
| vals.ai independent evaluation | vals.ai (third-party, preregistered) | Preregistered evaluation of twelve systems (2026-10-06): 400 human-audited claim-verification items built from SEC filings, plus a LegalBench slice. | Jev claim verification 0.975, tied with GPT-6-class at ~1/498th of Astra's cost, ECE 0.011 best of twelve — but LegalBench 0.730 last of twelve and one contract-NLI subtask at 0.485 | The criticism line's first citable third-party dataset: strongest and weakest results published side by side. Task-relative, not a model constant. |
| Jevman (Opper AI) | Opper AI; AGPL-3.0 on GitHub | 100 real-time Pac-Man games against the classic scripted ghosts; at each junction the model returns a probability per direction; answers over 2 seconds fall back to a logged backup rule; rankings are mean score with a 95% margin and every game replays exactly. | No scores quoted here — the board is live and we have not verified a snapshot; models within each other's margin count as tied. | A behavioral benchmark: decision quality in a dynamic game state instead of static labeled sets — a complement to accuracy boards, not a replacement. |
Read every board with three rules
Vendor board ≠ independent board
Liquid's Decision Index and Fastino's fast-decisions are house benchmarks: useful within their own family, never a neutral cross-vendor ranking. vals.ai is the current counter-example — preregistered and published with Jev's weakest results attached.
Different bench, different winner
On Fastino's board GLiNER2.5-Decide leads; on the independent JevBench decider-4b v2 does; on the Decision Index pplx-decider v1.1 has overtaken Jev; on ImageJevBench a one-person model leads. Whichever model “wins” depends on which board you open — including the fixtures on this page.
Boards drift — versions and small samples
JevBench has shipped multiple board versions where the same system's score moves; the Decision Index updates monthly; benchmarkheaven re-scores continuously; part of ImageJevBench's sealed set is synthetic. Compare boards only within one named version, and read small-N results as direction, not verdict.
Numbers marked “as captured” were read frame-by-frame from the linked walkthroughs. Board scores change over time, so treat every Jev benchmark result here as a snapshot and check the live version before quoting. This section last reviewed 2026-10-23.
Where these numbers come from
Image Decision Models guide
ImageJevBench v0.1.3 read frame by frame, plus Imajev-4B and Jev-Omni reviewing an unsigned W-8BEN with no OCR.
GLiNER2.5-Decide vs Jev
The Fastino scoreboard and the JevBench counter-ledger side by side — the “different bench, different winner” case study.
Liquid d1 local test
Decision Index v0.2.1 numbers versus the reviewer's own fanless-laptop re-runs.
PPLX Decider walkthrough
The Jev Decision Index hover cards, the 85.71% vs 84.51% vendor panel, and the v1.1 lead flip.
Decision model calibration
What ECE actually measures, and how vals.ai's best-of-twelve and worst-of-twelve calibration results fit one workflow.
Ollaya guide
benchmarkheaven quoted with its own caveats, the HN skeptics' section on stage, and every latency figure labeled vendor-reported.
Metrics Matrix
Four Core Evaluation Dimensions
Validating an AI decision model for live production requires simultaneous verification across these four vectors:
Macro F1 Score
Computes precision and recall independently per class before averaging, preventing high-volume queries from concealing critical failures on rare events.
P95 Tail Latency
Tracks 95th percentile response times under concurrent production load. Sub-400ms execution is critical for synchronous API gates and edge middleware.
Cost per 1k Inferences
Audits actual prompt and decision token burn per 1,000 inferences, ensuring scaled automation maintains decisive ROI over manual operations.
Confidence Calibration (ECE)
Measures honesty in probability scoring. When Jev assigns 0.90 confidence, empirical accuracy must closely track 90% to enable safe automation cutoffs.
Building an In-House Benchmark Suite
Follow these four engineering steps to calibrate criteria against your historical records:
Assemble a Gold Dataset
Curate 200–500 historic production samples with multi-pass human annotations, ensuring contentious edge cases have unambiguous ground truth.
Freeze Schema & Criteria Versions
Commit instructions and option criteria into git before evaluation. Avoid modifying descriptive text mid-test to maintain reproducible fixtures.
Log Full Inference Probabilities
Capture provider ID, model ID, raw state, selected choice, all option probabilities, runtime duration, and token usage for every test sample.
Plot Confusion Matrix & Threshold Curves
Graph the tradeoff between automation rate and accuracy across confidence gates (e.g., 0.70 to 0.95) to set your production cutoff threshold.
Frequently Asked Questions (FAQ)
Why doesn’t Jev rely on MMLU or GSM8K benchmarks?
MMLU and GSM8K assess encyclopedic knowledge and multi-step arithmetic, whereas Jev is an optimized decision engine (System One). In routing, content filtering, and policy enforcement, success depends on categorical boundary precision, calibrated probabilities, and sub-second latency.
Why is Macro F1 preferred over Overall Accuracy in decision models?
Production datasets suffer from severe class imbalances (e.g., 95% benign inquiries vs 5% critical disputes). An algorithm classifying everything as benign achieves 95% accuracy while completely failing business objectives. Macro F1 weights all classes equally.
What is Calibration Error and why does it dictate system safety?
Calibration tracks the alignment between model confidence and actual probability of correctness. If a model asserts 0.99 confidence on flawed choices, automated fallbacks fail. Well-calibrated probabilities allow engineers to safely automate high-confidence bands while diverting edge cases to human triage.
What should we do when two categories frequently cross-contaminate?
Confusion between two specific classes points to overlapping criteria definitions. Inspect misclassified samples, identify the differentiating operational factor, and inject explicit counter-clauses into both criteria definitions.
What standard governs benchmark publications on Jev 101?
We enforce a zero-fabrication standard. All official metrics require a committed public sample version, run date, verified provider endpoint, and runnable test script so teams can independently verify results.
Which Jev benchmark should I trust?
None of them blindly — trust comes from matching the question to the board. For a specific business decision, no published board beats running about 100 labeled examples of your own (our fixtures above show the workflow). For orientation: vals.ai is preregistered and independent; the Jev Decision Index and ImageJevBench are community-run with disclosed methodology; vendor boards (Liquid, Fastino) describe their own family best. Every board on this page publishes its own caveats — read those first.
Why do different Jev benchmarks crown different winners?
Because they measure different things on different data. Fastino's exact-match routing subsets favor its own encoder; JevBench's four-axis blend currently favors decider-4b v2; the Decision Index weights open reproductions across 43 benchmarks; ImageJevBench tests image inputs, where hosted Jev did not compete. The honest read is the one this page takes: quote the board, the version and the caveat together — and expect the winner to change.