Jev 101 logo
Jev 101

Guides / illustrated walkthrough

Jev Model Router: Build a Privacy-Gated LLM Router with Jev & OpenJev

Sam Witteveen builds a local model router powered by Jev: one typed call classifies every request, scores its difficulty, and gates on PII before choosing between a local MiniCPM5-2B, DeepSeek V4.1 Flash on OpenRouter, and a local Qwen-Image-2.1 image lane.

Quick takeaway

A model router is one endpoint that asks Jev three questions about every incoming prompt — a Choice for the lane (chit-chat, code, image), a Score for difficulty, and a Noul privacy gate — and routes accordingly: greetings stay on a local MiniCPM5-2B, code goes to DeepSeek V4.1 Flash on OpenRouter, images to a local Qwen-Image-2.1, and anything containing PII never leaves the machine. All three questions ride on the same state in a single call, so routing costs roughly the same as asking once (~120ms per decision; 88ms with a local Semif judge). In the demo, 40 of 53 requests (75%) were answered locally, avoiding $0.00813 in cloud spend against $0.00709 spent — and the router logs every decision to SQLite so you can audit exactly which model answered and why.

Video source

Sam Witteveen

19:03ZR7anrL50xs
Visit channelWatch original

Step-by-step walkthrough

  1. 1

    Set up two lanes first: one local workhorse, one cloud expert

    The router only needs two models to start. Run a small local model — Sam uses MiniCPM5-2B served on port 8000 through either LM Studio or Ollama — for chit-chat and light tasks, and wire one cloud model, DeepSeek V4.1 Flash via OpenRouter, for real coding work. Two optional lanes come later: a local Qwen-Image-2.1 for image generation and editing (released the day before recording, with a license that is effectively non-commercial), and Claude Opus 5 as an escalation lane for research-grade problems. The app UI exposes every lane as a toggle, so you can pin traffic to one model while testing.

    Jev model router slide listing the two starting lanes: MiniCPM5-2B served locally on port 8000 via LM Studio or Ollama, and DeepSeek in the cloud for complex code.
    Two lanes to start: MiniCPM5-2B locally, DeepSeek in the cloud.Watch at 1:10
  2. 2

    Replace parse-validate-retry with one typed call

    The whole router fits in a handful of if statements after a single call: ans = jev.ask(state, questions). If ans.task.choice equals "image", return qwen_image(prompt); if ans.hard.score is 4 or higher, return deepseek(prompt); if ans.pii.noul is above 0.7, keep it on local(prompt); otherwise fall through to minicpm(prompt). The slide marks what is NOT needed: parsing a {"task": "code"} string out of prose, validating it, and retrying on malformed output. Jev returns typed values with calibrated probabilities, so the routing logic reads like an if statement that understands language.

    Router pseudocode calling jev.ask(state, questions) then branching on ans.task.choice equals image, ans.hard.score over 4, and ans.pii.noul over 0.7, beside a NOT NEEDED card crossing out JSON parse, validate, and retry.
    One typed call, four if statements — the parse-validate-retry loop is gone.Watch at 2:26
  3. 3

    Ask Choice, Score, and Noul in one parallel call

    The router sends three questions on the same state: a Choice for the task lane, a Score for difficulty, and a Noul privacy flag. Jev evaluates them in parallel, so asking three questions costs about the same time as one — the slide budgets roughly 120ms per round trip, and one response carries all three answers: code 0.88, hard 8/10, pii 0.04. Just as important is the bottom banner: when confidence is low, do not guess — fall back to the safe default lane. Because every answer ships with a calibrated confidence, the router can route aggressively on high confidence and degrade gracefully everywhere else.

    Jev diagram showing one request with choice, score, and noul on the same state evaluated in parallel, returning one response with code 0.88, hard 8 out of 10, and pii 0.04, plus a banner reading when confidence is low fall back.
    Three questions, one round trip — and low confidence falls back instead of guessing.Watch at 5:32
  4. 4

    Write the privacy gate before you write the router

    The most interesting question is not "which model" but "where must this not go". The private Noul asks whether the conversation contains personal, confidential, client, financial, medical, or credential data — API keys, passwords, tokens, account numbers, private names and addresses — that should not leave this machine, with true meaning real secrets and false meaning generic content or obvious placeholders. A second Noul checks whether a correct answer needs fresh web data, so the web lane can be disabled whenever a prompt is private. Sam is explicit about the trust model: hosted Jev is trusted, frontier labs are not — and later in the video the judge itself is swapped to a local OpenJev so even the routing decision never leaks.

    Sent-to-the-judge JSON panel showing the private Noul that keeps personal, financial, medical, and credential data local, and the needs_web Noul that gates the web lane, with the answers panel returning code as the task choice.
    The private and needs_web Noul gates — PII pins the request to local lanes.Watch at 10:12
  5. 5

    Send a greeting and watch it stay local

    With the router running, "hey, how's it going" returns a lane banner: Chit-chat (100%), MiniCPM5-2B handles this locally — and the reply streams from the local model. The Next.js UI shows every lane as a row in the models panel (MiniCPM5-2B local, DeepSeek V4.1 Flash on OpenRouter, Qwen-Image-2.1 local, a web lane, Claude Opus 5 for hard tasks), and the totals card tracks requests, share answered locally, cost avoided versus spent, median judge latency, fallbacks, and private count. Asking for Python code flips the banner to DeepSeek; pasting a fake API key flips it straight back to local because the privacy gate fires.

    Jev Router chat app replying to hey how is it going with a MiniCPM5-2B local badge reading chit-chat 100 percent, beside a models panel of seven lanes and totals showing 41 requests with 33 answered locally.
    Chit-chat (100%): the greeting never leaves the machine — MiniCPM5-2B answers.Watch at 6:20
  6. 6

    Read the scoreboard, then move the judge itself on-prem

    After 53 requests the router has answered 40 (75%) fully locally, avoided $0.00813 of cloud cost against $0.00709 actually spent, with 6 fallbacks and 12 private requests — proof the savings exceed the judging overhead. The final move swaps the judge from hosted typesafe/jev-1.13 to Semif, an OpenJev clone running locally on Qwen3.5-4B: identical decisions, slightly different payload shape, and the whole pipeline — judge included — now runs offline. Sam's parting numbers: an entire video of demos cost under one cent, and with the privacy slider and per-lane thresholds exposed in Settings, tuning the router is a matter of reading your own SQLite decision log.

    Jev Router totals card showing 53 requests with 40 answered locally at 75 percent, cost avoided 0.00813 dollars against 0.00709 spent, median Semif judge at 318ms, next to a DeepSeek web-search answer citing npm and GitHub sources.
    75% answered locally, savings beating spend — with Semif judging entirely on-prem.Watch at 16:20

Frequently asked questions

What is a Jev model router?

A single endpoint that classifies every incoming prompt with the Jev model before any LLM runs. One call asks a Choice (which lane: chit-chat, code, image, reasoning), a Score (how hard is this), and up to two Nouls (is it private, does it need the web), then deterministic if-statements in your code dispatch the request to the matching model — a local small model for greetings, DeepSeek for code, an image model for art, and strictly local lanes for anything containing PII.

Why route at all instead of always using one big model?

Because most traffic is not hard. In the video, 40 of 53 requests (75%) were answered by the local MiniCPM5-2B, avoiding $0.00813 in cloud cost against $0.00709 spent on Jev judging and the remaining cloud calls. Routing also cuts latency for simple turns and lets you enforce privacy per request instead of trusting every vendor with every prompt.

Can the router run fully offline?

Yes. The final configuration swaps the judge from hosted typesafe/jev-1.13 to Semif — an OpenJev model running locally on Qwen3.5-4B through an OpenAI-compatible endpoint. Decisions take about 88ms and nothing leaves the machine: not the prompts, not the PII, and not even the routing decision itself.

What happens when the router is not confident?

It falls back instead of guessing. Every Jev answer carries a calibrated confidence, and the router's live rules send low-confidence tasks to the safe default lane — for example, task confidence below 0.6 escalates to the general model, privacy above the slider pins the request to local lanes, and outputs projected over 16,000 tokens jump straight to the bigger cloud model. The UI counts every fallback so you can audit misroutes in the SQLite decision log.

Related guides

Open Jev Models Landscape

Meet Semif and the other OpenJev clones you can use as an on-prem judge.

Read

Jev Pricing

Input-only billing is what makes one judge call per request economically boring.

Read

Jev API Reference

The evaluate endpoint, typed questions, and probability payload behind jev.ask.

Read

More video walkthroughs