Jev 101 logo
Jev 101

Guides / illustrated walkthrough

When to Use Jev: An Engineer's Audit of Claims, Gates, and Failure Modes

The Stack audits Jev for engineering teams: verifying the 193x/444x vendor claims, the allow/ask-a-human/block command gate, calibration curves, six failure modes, and a shadow-mode path to production.

Quick takeaway

Jev is a decision engine, not a chatbot: your code sends text plus a question whose answers are fixed in advance — a Choice from a list, a binary Noul, or a numeric Score — and gets a typed answer with a probability back in as little as 0.114 seconds. The video stress-tests the marketing: the headline 193.6x faster / 444.6x cheaper figures come from TypeSafe's own West-Coast-laptop evals, while an independent 2,000-email benchmark measured 239ms vs 687ms against Claude Haiku 4.5 (about 3x faster, not 200x) and roughly 4 vs 46 cents per 1,000 emails (about 12x cheaper). The zero-hallucination claim is a schema guarantee, not an accuracy promise: on a broad phishing question Jev scored 62.6% correct versus Haiku's 81.3%, but split into five narrow signals and weighted on 1,000 labeled emails it reached about 95% versus 93%. The safe adoption path is shadow mode first, confidence thresholds second, hardcoded rules always.

Video source

The Stack

15:57mJnOdnOrh9A
Visit channelWatch original

Step-by-step walkthrough

  1. 1

    Start with the gate: allow, ask a human, or block

    The recurring example is a scheduling agent that wants to run rm -rf ./project. Rules in code cannot tell safe cleanup from disaster, and a chat model asked for a one-word verdict may return three paragraphs explaining its feelings. Jev removes the chat box: your code sends the command text plus a question whose answers are locked in advance — allow, ask a human, or block — and gets back one of those exact options with a probability score, so an ordinary if statement can act immediately. In the LangChain agent harness published two days after launch, this pattern is Jev's second job: reviewing every tool call an autonomous agent attempts and stopping risky ones before execution. Type-safe's own Doom demo ran on the same trick — a move chosen in 0.114 seconds versus 8.566 seconds for GPT-5.6 Terra.

    Diagram of a coding agent piping a rm -rf ./project command through Jev, which returns block from an allow / ask a human / block menu labeled only these, beside an if statement that calls run(cmd) or stop(cmd).
    The gate: one rm -rf command, three allowed answers.Watch at 2:28
  2. 2

    Three question types: Choice, Noul, and Score

    Every question you write falls into one of three categories. A Choice selects one option from your fixed list — allow, ask a human, block — and returns probabilities for all of them. A Noul, a strangely named binary check, asks a yes-or-no question and returns a single probability between zero and one, like is this destructive. A Score rates the input along a numerical scale you define, such as how risky is this command from 1 to 5. Note what never happens anywhere in the cycle: Jev never writes a word of explanation. Standard models pay for token-by-token generation like a painfully slow typist charging per syllable; Jev skips generation entirely and evaluates all responses in parallel in a single pass. As developer Flavio Copes put it, it works like a smart if statement built into your software — a decision engine called by code, not a conversationalist.

    One Jev call branches into three panels: a Choice asking should this command run with three options, a Noul gauge reading is this destructive from no 0 to 1 yes, and a Score scale rating command risk from 1 to 5.
    Choice picks from a list, Noul answers yes/no, Score rates 1-5.Watch at 3:08
  3. 3

    Audit the 193x and 444x claims before you believe them

    On paper Jev lists at 4.2 cents per million input tokens with replies free — a reseller listing puts it near 5.8 cents — and TypeSafe quotes end-to-end responses of 70ms-500ms, claiming 40x-200x speedups with peaks of 193.6x faster and 444.6x cheaper. Read the footnotes. TypeSafe's launch notes admit it cannot prove the service is not subsidized, that its speed tests ran from the team's West Coast laptops, and that its evals were written by its own researchers, so local bias exists. When an independent developer in France benchmarked Jev against Claude Haiku 4.5 on 2,000 emails, Jev took 239 milliseconds per decision versus 687 — about three times faster, not 200 — and cost roughly 4 cents per 1,000 emails versus 46, about 12 times cheaper. A mixed DSPy pipeline that moved only its yes/no and pick-one steps to Jev gained 16% speed and 30% cost.

    Launch-claim cards highlighting that end-to-end response time is 70ms-500ms and that the 193.6x faster, 444.6x cheaper figures sit on the higher end of real-world gains, above a best-case scale weighing Jev against an Anthropic model.
    Even TypeSafe files 193.6x and 444.6x under best case.Watch at 7:52
  4. 4

    Narrow checks and a weighted sum beat one broad prompt

    The same independent benchmark exposed the accuracy gap. Asked one broad question — is this email phishing or legitimate — Jev was correct 62.6% of the time while Claude Haiku 4.5 hit 81.3% on the identical prompt, so a cheap general chat model easily won the unassisted decision. Jev caught up when the developer stopped treating it like an oracle and split the problem into five narrow technical signals — sender domain versus claimed brand, artificial urgency, and similar markers shown as sig_ gauges — then fit weights on 1,000 labeled training emails and combined the signals with a weighted sum. Combined accuracy jumped to about 95%, with Haiku at about 93% using the same approach: a technical tie. The lesson is that Jev excels on narrow, well-defined checks, but 95% required labeled examples and a scoring setup you build and maintain yourself.

    Five phishing signal gauges from sig_domain_mismatch to sig_generic_sender feed a weighted sum tuned on 1,000 labeled emails, with accuracy bars showing Jev at 95 percent versus Haiku 4.5 at 93 percent, marked tie.
    Five narrow signals plus a weighted sum lift Jev to 95%.Watch at 11:48
  5. 5

    Calibration and the six documented failure modes

    The zero-hallucination promise is narrower than it sounds: Jev physically cannot emit text outside your fixed answer list — 2,000 independent calls produced zero malformed responses — but it can absolutely pick the wrong option, and schema-valid is not the same as correct. TypeSafe calibrates Jev so a 90%-confidence answer should be right nine times in ten, yet in independent email testing Haiku's confidence tracked reality more closely: ECE 0.097 versus Jev's 0.134 on the video's calibration chart. The official benchmarks also measure agreement with frontier models like GPT-6 Astra and Fable 5.1 rather than verified truth. The Jev 1.13 documentation lists six failure modes: literal reading, math and numbers, date and time comparison, indirection, large states full of irrelevant detail, and adversarial content — a file arguing that deleting the folder is safe can nudge the gate toward allow.

    Calibration chart under a 90 percent sure banner plotting observed share of phishing against predicted probability, where Jev scores ECE 0.134 and claude-haiku-4.5 scores ECE 0.097 against the perfect calibration line.
    On confidence, Haiku 4.5 tracks reality more closely than Jev.Watch at 12:08
  6. 6

    Roll out in shadow mode, then gate on confidence

    The safe adoption path starts with shadow mode: run Jev beside your current workflow, let it quietly evaluate real traffic on decisions where you already know the absolute truth, and compare answers before handing over control. Then use the reported confidence scores to set thresholds — high-confidence answers pass straight through, borderline cases escalate to a person or a more powerful model, and destructive actions stay denied by strict rules in your own code. In the recurring example, Jev instantly authorizes common read commands like cat notes.md, folder deletions route to manual approval, and a full root wipe is blocked by hardcoded shell checks regardless of what Jev says. Keep questions specific, remember Jev is still in early access under a waitlist with only days of public data, and treat it as a decision maker, not a code writer — test before trusting it with anything mission-critical.

    Real traffic keeps flowing through current logic to action while a quiet copy runs Jev in parallel, and a known-answers card tracks checks and crosses to show how often the two answers agree.
    Shadow mode first: let Jev listen beside the workflow before it acts.Watch at 14:53

Frequently asked questions

When should you actually use Jev?

When your software constantly makes small, repetitive decisions where every possible response is predetermined: classifying support tickets, categorizing exceptions, routing requests to the right model tier, or screening every tool call an AI agent attempts. It is the wrong tool for chatting with customers, writing text or code, arithmetic, counting, and date comparisons — and it should never be the sole security control, only a smart first filter alongside strict rules in your own code.

Is Jev really 193x faster and 444x cheaper than big models?

Those are TypeSafe best-case numbers from its own evals, run on the team's West Coast laptops with tests its own researchers wrote. Independent testing against Claude Haiku 4.5 on 2,000 emails measured 239ms per decision versus 687ms — about 3x faster — and about 4 versus 46 cents per 1,000 emails, roughly 12x cheaper. A mixed DSPy pipeline that only moved its binary and choice steps to Jev became 16% faster and 30% cheaper.

Can Jev hallucinate?

Not in the schema sense: it physically lacks the mechanism to produce text outside your fixed answer list, and 2,000 independent calls returned zero malformed responses. But schema-valid is not correct — on a broad phishing-or-legitimate question Jev was right 62.6% of the time versus Haiku 4.5's 81.3%. Split into five narrow signals with weights fit on 1,000 labeled emails, Jev reached about 95% and Haiku about 93%.

What are Jev’s documented failure modes?

The TypeSafe documentation for Jev 1.13 lists six: literal reading (it evaluates the text you sent, not what you meant), math and numbers (counting and arithmetic are unreliable), date and time comparison, indirection, large states full of irrelevant detail, and adversarial content — persuasive text inside an evaluated file, such as an argument that deleting a folder is safe, can push the decision toward allow.

How do you roll Jev out safely in production?

Run it in shadow mode first: let it evaluate real traffic beside your current stack on decisions where you already know the truth, and measure agreement before giving it control. Then set thresholds on its confidence scores — high-confidence answers flow straight through, borderline cases escalate to a human or a bigger model — and keep hardcoded rules for destructive actions so a root wipe is blocked no matter what any model says.

Related guides

Jev vs LLM Guide

A deeper head-to-head on where a decision engine beats a chat model — and where it never will.

Read

LLM Fallback Strategy

Confidence-gated fallbacks are exactly how low-risk Jev answers pass and doubtful ones escalate.

Read

Jev Benchmarks

Independent latency, cost, and accuracy numbers to compare against vendor best-case claims.

Read

What Is Jev?

The input-only pricing and parallel evaluation that make fractions-of-a-cent decisions possible.

Read

More video walkthroughs