Jev 101 logo
Jev 101

Guides / illustrated walkthrough

Microsoft Decision-1 Tested: 146 Independent Tests, a 93% Mean, 124/125 When Confident — Every Number Frame-Checked

A hands-on Microsoft Decision-1 review turned into a step-by-step page: the regular-LLM-versus-decision-model explainer card, the five-week Jev-to-Microsoft timeline, the vendor-reported claims kept at arm’s length, the 146-test harness across 17 sets run entirely through OpenRouter, the 93% mean scorecard with ten perfect sets, the judgment sets where it slipped, the calibration pair (124/125 above 90% confidence, 2/7 below 70%), 180 rewrites with zero flips, the 32,768-token context wall, and a one-cent total bill — every on-screen number verified frame by frame.

Quick takeaway

The video in one pass: Microsoft Decision-1 is a never-writes-a-sentence decision model — read the state once, return a calibrated probability per allowed answer, one forward pass, zero generated tokens — released October 9, 2026, priced at $0.042 per 1M input tokens with output free (OpenRouter: microsoft/microsoft-decision-1; also on Azure Foundry, which this test deliberately skipped). Category card, from Microsoft’s own docs: regular LLM = input → writes token by token → “I think this is probably a billing issue because…”; decision model = input + allowed answers → one pass → billing 0.997 · technical 0.001 · account 0.001. Identity per the video: base is Alibaba’s open Qwen3.5 (9B parameters) retrained for single-pass evaluation; one size, text-only; Microsoft says it will later rebase the recipe onto its own and OpenAI models, and it evaluates itself on the Jev benchmark. Timeline card: Jev/TypeSafe (Sep) → Laya/open source (Sep) → Clef/Cloudflare (Oct 1) → Decider v1.1/Perplexity (Oct 6) → Decisions API/OpenAI (Oct 6) → d1/Liquid AI (Oct 7) → Decision-1/Microsoft (Oct 9). Vendor-reported claims, kept at arm’s length: top accuracy on 36 benchmarks with ~150,000 held-out questions; median latency 35× faster than GPT-6 Sol; 200× cheaper than GPT-6 Sol on 10,000+ Xbox feedback items at similar quality; 1.3% answer flips on reworded or reordered requests. The independent test: 146 cases across 17 sets, each with a known correct answer, plus 36 long-document calls and 180 perturbation requests, all via OpenRouter from Bengaluru (12 Oct 2026). Scorecard: 93% mean; ten sets at 100% (ticket routing, gamer feedback themes, phishing emails, prompt injection, content moderation, JSON validation, multilingual across 10 languages plus code-mixed Hinglish — “Bro, OTP is not coming at all” — code vulnerabilities, fraud on CSV rows, HTML page type); model routing 90%, bug severity 88%, multi-question call 88%, LLM-as-judge 81%, log triage 81%, agent guardrail 80%, search relevance 75%. Failure mode: all misses are judgment calls, and the quoted errors lean cautious — approval requested before a harmless code fix, a true additional fact flagged as a hallucination. Calibration (the video’s most important part): 124/125 right when over 90% confident, 2/7 right when under 70%, calibration error 0.025 — a usable gate: act when confident, escalate to a human or bigger model otherwise. Robustness: 180 rewrites/reorderings/renamings/space-injections/JSON-wrappings, zero flips — Microsoft’s 1.3% is conservative on this data. Context: clause found every time up to ~19K tokens; refused at ~33K; hard limit 32,768 tokens including the decision token (HTTP 422) — split long inputs. Speed: median ~620 ms via OpenRouter to Azure vs ~230 ms on OpenRouter’s page — network dominates. Cost: the whole experiment, about one US cent. Takeaway: not a chatbot — a management layer for agents that directs, labels, evaluates and blocks; the “knows when it isn’t sure” property is the part you need before letting agents act on their own.

Video source

AI WITH Rithesh

5:29qzPv-IrybQM
Visit channelWatch original

Step-by-step walkthrough

  1. 1

    What Decision-1 is: a model that never writes a sentence

    The video opens with the category in one card, sourced from Microsoft’s own documentation. A regular LLM takes the input and writes its answer token by token: “I think this is probably a billing issue because…” — prose you then have to parse. A decision model takes the input plus a set of allowed answers and makes one pass, returning a probability for each: billing 0.997, technical 0.001, account 0.001. You can ask a yes/no question, offer a list of options, or request a rating on a scale — the code receives a number and acts on it: continue if the model is confident, hand the task to a human if not. Pricing is the other headline: $0.042 per million input tokens, output free. And a naming note worth one sentence: Decision-1 is a model name with a hyphen — not a claim to being the first decision model, as the timeline in the next step makes clear.

    Microsoft Decision-1 explainer card contrasting a regular LLM that writes token by token with a decision model that returns billing 0.997, technical 0.001 and account 0.001 in one pass
    One pass, three probabilities, zero sentences — the category definition on Microsoft’s own docs card.Watch at 0:45
  2. 2

    From TypeSafe to Microsoft: the five-week wave

    The idea started with a small startup — TypeSafe’s Jev — and the timeline card shows how fast the rest moved: Laya (open source) in September, Clef from Cloudflare on October 1, Decider v1.1 from Perplexity and the Decisions API from OpenAI on October 6, d1 from Liquid AI on October 7, and Decision-1 from Microsoft on October 9. Microsoft, the card notes, even evaluates itself using the Jev benchmark. Under the hood, per the video: Alibaba’s open Qwen3.5, a 9-billion-parameter model that Microsoft retrained to evaluate responses in a single pass — and Microsoft says it will later transfer the recipe to its own and OpenAI models. Two scope facts matter for planning: there is currently one size only, and it only reads text.

    Timeline of decision-model launches from September to October 9 2026 showing Jev by TypeSafe, Laya open source, Clef by Cloudflare, Decider v1.1 by Perplexity, Decisions API by OpenAI, d1 by Liquid AI and Decision-1 by Microsoft
    Five weeks from one startup’s idea to a Microsoft launch — the card names every stop.Watch at 1:06
  3. 3

    The vendor claims, kept at arm’s length

    Before running anything, the video puts Microsoft’s numbers on screen — with a label that does exactly what it should: “Microsoft says / vendor-reported”. The claims: highest accuracy on 36 benchmarks with nearly 150,000 held-out questions the model never studied; median response speed 35× faster than GPT-6 Sol; the Xbox team sorted 10,000+ player feedback items at GPT-6-Sol-level quality for about 200× less; and when the wording or order of a request changes, the answer flips in only 1.3% of cases. These are Microsoft’s numbers, measured on Microsoft’s terms — the rest of the video is the reviewer finding out what they look like on his own tests, which is the only part worth copying into your own decision.

    Vendor-reported card from Microsoft claiming Decision-1 is 200 times cheaper than GPT-6 Sol on more than 10,000 Xbox feedback items at similar quality
    “Microsoft says, vendor-reported” — the label the rest of this page stress-tests.Watch at 1:53
  4. 4

    The harness: 146 tests in 17 sets, no Azure required

    The reviewer’s setup fits on three cards. 146 test cases across 17 sets, each with a known correct answer written in advance — support-ticket-style routing, phishing emails, logs, code, CSV rows, web pages, ten languages. 36 long-document calls, with one clause hidden at the top, middle or end. And 180 reworded or reordered requests for the robustness check. Two practical choices make this reproducible by anyone: everything ran through OpenRouter (model id microsoft/microsoft-decision-1), so there was no Azure configuration at all, and the whole thing ran on his laptop in Bengaluru — the source line on every card dates the run: 12 Oct 2026. Keep the scale in mind as you read the scores: 146 graded cases is a solid field test, not a benchmark suite.

    Three stat cards describing a 146-test harness across 17 sets with known right answers, 36 long-document calls with a clause hidden at top middle and end, and 180 reworded or reordered requests
    The whole experiment on three cards: 146 scored cases, 36 long-document probes, 180 perturbations.Watch at 2:28
  5. 5

    The scorecard: 93% mean, 10 of 17 sets perfect

    The headline chart needs no squinting: mean 93%, with ten of the seventeen sets at 100% — ticket routing, gamer feedback themes, phishing emails, prompt injection, content moderation, JSON validation, multilingual, code vulnerabilities, fraud on CSV rows, and HTML page type. The multilingual set deserves its own sentence: ten languages plus code-mixed input — Hindi, Kannada, Tamil, Japanese, Arabic, and Hinglish like “Bro, OTP is not coming at all” — all perfect. On camera, a French “I’d like an invoice for last month’s payment” routes to billing at 99.9% and a German “my package is marked delivered but never arrived” routes to shipping at 99.9%. Below the green bars: model routing 90%, bug severity 88%, multi-question call 88%, and the four judgment sets you’ll meet in the next step.

    Bar chart of Microsoft Decision-1 accuracy by data type showing a mean of 93 percent with ten sets at 100 percent including multilingual, phishing emails and prompt injection, and search relevance lowest at 75 percent
    Ten green bars at 100% — and the multilingual row is the one Microsoft’s launch barely mentioned.Watch at 2:33
  6. 6

    Where it slipped: the judgment sets

    Every miss is a matter of judgment rather than fact: search relevance 75%, agent guardrail 80%, LLM-as-judge 81%, log triage 81%. The two quoted error cases are more instructive than the percentages. First, the model wanted approval before making a harmless code fix — refusing to act without a human sign-off that wasn’t needed. Second, it called a true additional fact a hallucination — flagging correct new information as fabricated. The reviewer’s read is the right one: when unsure, the model acts cautiously, and for a protective barrier that is exactly the bias you want. A guardrail that sometimes stops work it should have allowed costs minutes; one that waves through destructive actions costs the incident.

    Where it slipped card listing Microsoft Decision-1 test scores of 75 percent on search relevance, 80 percent on agent guardrail, 81 percent on LLM-as-judge and 81 percent on log triage
    Every miss is a judgment call — and both error examples err on the cautious side.Watch at 3:09
  7. 7

    Calibration: the part that matters most

    This is the section the reviewer calls the most important, and the one worth building on. Whenever the model said it was over 90% confident, it was right on 124 of 125 answers. Whenever it was under 70% confident, it was right on only 2 of 7. The calibration error across the run: 0.025, where lower is better. The operational translation: the confidence indicator is actually usable — let the model act when it is confident, and send everything else to a person or a larger model. That is the property that turns a decision model from a classifier into infrastructure: the number it returns tells you what to do with the number it returned. Standard honesty applies — 132 graded answers is a small, self-made sample, so verify the gate on your own labels before wiring it into production.

    Calibration cards for Microsoft Decision-1 showing 124 of 125 correct answers when over 90 percent sure, 2 of 7 when under 70 percent sure, and a 0.025 calibration error where lower is better
    Confidence that means something: near-perfect above 90%, barely a coin flip below 70%.Watch at 3:30
  8. 8

    180 rewrites with zero flips, and the 32,768-token wall

    Two stress tests round out the field report. Robustness: shuffle the options, change their order, rename them to A/B/C, rephrase the question, add random spaces, wrap the whole text in JSON — 180 requests in total, and not a single answer changed. Microsoft’s claimed 1.3% flip rate looks conservative on this dataset, in the good direction. Long documents: the reviewer hid one clause about pets in a lease agreement, rotating its position between the beginning, middle and end. Up to about 19,000 tokens, the model found the clause every single time. At around 33,000 tokens the API simply refused — and the error card names the ceiling exactly: the question prompt exceeds the 32,768-token context limit including the decision token. If your inputs run long, split them before you call.

    HTTP 422 error card from Microsoft Decision-1 stating the question prompt exceeds the 32768-token context limit including the decision token
    The hard ceiling, in the API’s own words: 32,768 tokens including the decision token.Watch at 4:32
  9. 9

    Speed, cost, and the management-layer takeaway

    The latency reconciliation is refreshingly honest: the reviewer’s median was around 620 ms from Bengaluru via OpenRouter to Azure, while OpenRouter’s own page lists about 230 ms — most of the waiting was network, not model. The model page itself confirms the economics: $0.042 per 1M input tokens, $0 output, 32K context, released Oct 9, 2026. The cost of the entire experiment — 146 scored tests, 36 long-document calls, 180 perturbations — came to about one US cent. The closing argument is the framing to keep: don’t think of Decision-1 as a chatbot; think of it as a management layer for your agents — something that directs, labels, evaluates and blocks. In his tests it was accurate, unbothered by rephrasing, and — most importantly — it knew when it wasn’t sure. That is exactly what you want in place before letting an agent act on its own.

    OpenRouter model page for microsoft/microsoft-decision-1 listing 0.042 dollars per 1M input tokens with free output, 32K context, released October 9 2026, beside a playground agent guardrail example
    The reviewer’s actual pipe: OpenRouter at $0.042 per million input tokens, output free — a whole evaluation for about a cent.Watch at 4:52

Frequently asked questions

What is Microsoft Decision-1?

Microsoft Decision-1 is a decision model released by Microsoft on October 9, 2026: instead of generating text, it reads an input once and returns a calibrated probability for each allowed answer — a yes/no verdict, a pick from your option list, or a score on your scale — so applications can act, defer or ask for review based on the number. Per the video, it is post-trained from Alibaba’s open Qwen3.5 (9B parameters), currently ships in one size, and only reads text; Microsoft says it plans to transfer the approach to its own and OpenAI models, and it evaluates the model on the Jev benchmark. Pricing is $0.042 per 1M input tokens with output free, served both on Azure Foundry and via OpenRouter as microsoft/microsoft-decision-1. One disambiguation: Decision-1 is a product name — not a claim to being “the first decision model”; it arrived last in a five-week wave that started with TypeSafe’s Jev.

How accurate is Decision-1 really?

Two different answers, and the video keeps them separate. Microsoft’s vendor-reported claims: highest accuracy on 36 benchmarks with ~150,000 held-out questions, 35× faster median latency than GPT-6 Sol, and 200× cheaper than GPT-6 Sol on 10,000+ Xbox feedback items at similar quality. The independent run in this video: a 93% mean across 146 self-written tests in 17 sets, with ten sets at 100% — including phishing emails, prompt injection, code vulnerabilities, CSV fraud detection, HTML page typing, and a multilingual set spanning Hindi, Kannada, Tamil, Japanese, Arabic and Hinglish. The misses concentrated in judgment-heavy sets: search relevance 75%, agent guardrail 80%, LLM-as-judge 81%, log triage 81%. Both number sets are honest as long as they stay labeled — vendor benchmarks on one side, a small self-made test set on the other.

Do you need an Azure account to try Decision-1?

No — that is one of the video’s practical points. Everything ran through OpenRouter under the model id microsoft/microsoft-decision-1, with no Azure configuration at all, at $0.042 per 1M input tokens and free output; the reviewer’s entire experiment cost about one US cent. Azure Foundry is the first-party route (Microsoft’s own docs carry the model card and the decision/LLM contrast the video quotes), but this test never touched it. Two constraints to plan around either way: the context limit is 32,768 tokens including the decision token, and the model only reads text — one size, no image or audio input.

What is Decision-1’s context limit?

32,768 tokens, and the decision token counts toward it — the API returns HTTP 422 with the message “Question prompt exceeds the 32768-token context limit including the decision token” when you cross it. The video maps the boundary from both sides: a clause hidden at the beginning, middle or end of a lease agreement was found every time up to about 19,000 tokens, while a ~33,000-token input was refused outright. The fix is the obvious one — split long documents into chunks and ask per chunk. Also remember the model is text-only, so the limit applies to the state plus your question text, with no images to spend it on.

Can you trust Decision-1’s confidence score?

In this run, more than in most — with caveats. Above 90% confidence the model was right 124/125 times; below 70% confidence it was right only 2/7; calibration error for the whole run was 0.025 (lower is better). That spread is what makes the score operationally useful as a gate: let the model act when it is confident, and route low-confidence calls to a human or a larger model. The caveats matter though: 132 graded answers on the reviewer’s own labels is a small single-run sample, and calibration measured on one workload does not automatically transfer to yours. Treat the 124/125 as a strong signal worth testing on your cases — not as a specification.

How does Microsoft Decision-1 compare to Jev?

They are the same category from different vendors — and per the video’s timeline, Microsoft is the follower. Jev is TypeSafe’s hosted decision API, the product that started the decision-model wave in September; Decision-1 is Microsoft’s October 9 entry, post-trained from Alibaba’s open Qwen3.5 9B, and Microsoft even evaluates it on the Jev benchmark rather than inventing a new one. No head-to-head numbers between the two exist in this video, so treat this page as the independent field report on Microsoft’s model and compare methodology on our other pages: the four-arm JEV-27B local test, the decision-model API hub for the category panorama, and the benchmark-criticism guide for how to read vendor claims like the 36-benchmark, 35× and 200× numbers above.

Related guides

AutoTrust JEV-27B Tested Locally: 72-Case Results

The independent-evaluation sibling: another uploader-built test harness scoring a decision model, complete with a failed confidence gate and a published flaw in the test itself.

Read

Decision Model API Hub

The category panorama — Jev, Decider, d1, Decisions API and now Decision-1 side by side across the same axes this page tested.

Read

Jev vs OpenAI Decisions API

OpenAI’s entry from the same timeline card, compared against the hosted Jev API that started the wave.

Read

Jev vs Luna Benchmark Guide

How to read vendor benchmark claims like the 36-benchmark, 35× and 200× numbers above — the methodology-criticism axis.

Read

Liquid d1 Guide

Liquid AI’s open-weight d1 family from the same five-week timeline — the run-it-locally counterpoint to Microsoft’s hosted model.

Read

More video walkthroughs