AutoTrust JEV-27B Tested Locally: Four Arms, 72 Cases, and One 0.969 Score That Was Wrong

This is RUNTIME.'s independent local evaluation of AutoTrust JEV-27B (Hugging Face autotrust/JEV-27B, Apache-2.0) — a 10:34 field test that treats the model's official claims as things to verify rather than repeat. First, identity, because the 27B Jev-family space now has two players: AutoTrust AI Lab's JEV-27B is a finetune of Alibaba's Qwen3.8-27B that distills decision behavior from TypeSafe's hosted Jev (the HF card calls it a student of TypeSafe Jev 1.13) by freezing the backbone and training roughly 109 million parameters — a decision adapter plus a scoring head; the local release bundles those additions, so it never calls the teacher. It is not the same thing as ZefanCai/Open-Jev-27B-v1.1, a separate community release shipped as a PEFT adapter package (adapter plus head.pt) on the same Qwen3.8-27B base — different publisher, different artifact — and AutoTrust's own JEV-27B-VL is the vision variant of this model, not the one tested here. The official claims on the slide: about 137 ms per decision on a B200, roughly half the hosted TypeSafe Jev API's published 238-301 ms (which includes network time), with coding ability unchanged — 78.0% versus 78.0% on 128 of 164 tasks. The test rig: a DGX Spark (GB10) running the pinned revision in 16-bit BF16 — explicitly not Q4 or Q8 — with two accelerated components unavailable and fallback implementations in their place, so none of this is an optimized throughput benchmark. Four arms see identical inputs (a written policy, the customer message, the same options list — Billing/technical, Security/ask): the rules control is a deliberately small deterministic parser; the decision head is the trained typed-choice path; the ordinary answer runs the same backbone with the adapter off, thinking off and a strict structured format; the reasoning path generates thinking first, then scores. Protocol: 24 development cases exist to test one thing — whether a confidence score could gate when to escalate to reasoning — then 72 held-out cases are scored, and repeats (reordered options, irrelevant details, misleading instructions) check consistency only, never extra independent problems; no LLM judges the answers, the cases were written and checked in advance, and a high score never overrides a wrong action. Case 1 shows where interpretation earns its place: a duplicate charge from yesterday that was refunded, plus an export failure today, routes to technical support — the rules control matched the word 'charge' and picked billing, while quick head, ordinary generation and reasoning all read the change in state correctly. Case 2 shows where ordered conditions bite: retry is authorized, severity is normal, three of four attempts are used, but the diagnostic is 14 minutes old against a 13-minute freshness limit — the required action is collect; rules and reasoning get it, quick head and ordinary generation both retry, and the reasoning answer cost 85.94 s against the quick answer's 0.66 s in that setup (a related critical-severity variant requires escalate first; generation still retries). The 72-case scoreboard: ordinary generation 50, quick head 63, reasoning 71, rules 70 — and moving from quick head to reasoning fixed eight answers without flipping a single correct one wrong in this run, while the tiny rules control finished one point behind reasoning. Then the honesty sections, which are the reason to watch. First, the testers report a flaw in their own test: five routing cases contain a policy exception that sends a billing issue to Security for a human audit while the Security option itself describes unauthorized account activity — the expected route follows the explicit exception, but the wording pulls against it. They keep the original scores, then re-run without the five: quick head 63/67, reasoning 67/67, rules 65/67, generation 50/67 — a sensitivity check, not a new benchmark, on a small synthetic set with an ordinary-generation control that never got a prompt search. Second, the confidence shortcut fails: six thresholds from 0.50 to 0.95 were tried on the 24 development cases and none met the accuracy requirement, so the video contains no working hybrid gate and no held-out result for one. The clean proof is a single development case where the quick head chose retry — the required action was collect — with a score of 0.969: even the highest candidate threshold, 0.950, would have accepted it without reasoning. The score describes the model's preference among the supplied choices; reading 0.97 as a 97% chance of correctness needs evidence that predicted confidence matches observed accuracy across many representative examples, and picking a new threshold after inspecting answers is just a new experiment (new design, separate development examples, freeze, fresh held-out cases). Third, the runtime reality: median end-to-end request time was 0.65 s for the quick head (p95 0.72 s), 2.34 s for ordinary generation (p95 2.60 s) and 102.20 s for reasoning (p95 128.50 s), one request at a time; reasoning hit its 512-token limit on 20 of the 72 cases and was forced to read out an answer — a correct final choice does not mean the reasoning completed; the model files weigh about 50 GiB with the highest sampled GPU process around 52 GiB, and because the Spark uses unified memory those two overlap rather than add. The fast decision head changes how you get the answer; it does not turn a 27-billion-parameter model into a small download. Finally, 96 presentation variants built from 24 parent cases — reversed options, permutations, irrelevant detail, misleading instructions planted in the customer text where they are data, not policy — left the semantic choice unchanged 91/96 times for the quick head, 95/96 for reasoning, 91/96 for generation and 96/96 for rules, which is consistency, not correctness: a system can repeat the same wrong answer every time, and a few misleading messages are not a security guarantee. The video's closing advice is the operational summary: the quick head is interesting when an agent picks from a known set and the wording needs interpretation; keep clear conditions in code so the model interprets the message while the program checks diagnostic age and permission, act only when they agree, and treat disagreements as concrete things to inspect — then try another backend or a better test, and share prompts, settings and results.