What is an LLM fallback strategy?

An LLM fallback strategy defines what a production system does when an answer cannot be trusted: retry transport errors with a bounded budget, fall back to a stronger model or an alternate path when confidence is borderline, and escalate to a human or a safe default when confidence is low or the action is irreversible. A confidence-gated chain makes each hop explicit, measurable, and auditable in code.

When should you retry, fall back, or escalate to a human?

Retry only infrastructure failures — 5xx, timeouts, rate limits — with a bounded budget; the answer never changes between identical attempts. Fall back when the answer is well-formed but uncertain: calibrated confidence below your automation threshold triggers a re-ask with a stronger model or an alternate path. Escalate to a human when the action is irreversible (payments, deletions, outbound email) or when confidence stays below threshold after escalation.

How do you choose a confidence threshold for routing?

Derive it from a calibration curve on labeled samples, not intuition: plot predicted confidence against observed accuracy, then set the auto-execute cutoff where precision meets the cost of a wrong automated action. With calibrated probabilities — Jev trains with RLCD, so 0.90 confidence means roughly 90% actual accuracy — the threshold becomes an accuracy contract. Run the policy in shadow mode (log-only) before enforcing it.

What is confidence threshold routing?

Confidence threshold routing sends each request down a different execution lane based on the model's calibrated confidence: high-confidence traffic executes automatically (≥ 0.85 is a common default), the 0.60–0.85 band re-asks a stronger model, and sub-0.60 traffic reaches a human queue. It turns "the model was unsure" from a log line into a traffic split you can monitor, alert on, and tune.

When does a confidence-gated fallback chain not work?

Three honest limits. It cannot supervise context it never sees — the evaluation state must contain everything the decision depends on. It must not be the only gate for irreversible operations: a human or a deterministic rule keeps final ownership regardless of confidence. And calibration drifts with distribution shift, so re-validate thresholds quarterly on fresh labeled samples instead of trusting them forever.

Jev recipe / reliability engineering

LLM Fallback Strategy: The Confidence-Gated Chain

Reliability engineering for LLM systems is more than retry. Gate every answer on calibrated confidence: auto-execute when high, re-ask a stronger model when borderline, escalate to a human when low.

Build the confidence-gated fallback chain in six steps

01Separate the vocabulary first: retries answer infrastructure faults (5xx, timeouts), fallbacks answer untrusted answers, escalations answer irreversible consequences. Most fallback logic ships broken because all three share one code path.

02Gate automation on calibrated confidence only — 0.85 for auto-execute. A threshold is meaningless unless a 0.9-confidence answer is right about 90% of the time; Jev probabilities are calibrated by RLCD training so that identity holds.

03Design the middle lane before the edge cases: the 0.60–0.85 band re-asks the same decision to a stronger model once, and only the sub-0.60 tail reaches a human queue.

04Write the decision table in application code: retry transport errors with a bounded budget, fall back on confidence, escalate to a human on high stakes or irreversibility — never let the model own the policy.

05Add hysteresis and shadow mode: widen the middle band for flip-prone inputs, and run the chain log-only against production traffic before you let it enforce.

06Log every hop — answer, confidence, lane — and recalibrate thresholds from real outcomes each quarter instead of folklore.

schema / fallback decision contract

{
  "fallback_action": {
    "type": "choice",
    "instructions": "Decide the next action for this proposed answer",
    "criteria": {
      "execute": "High confidence and reversible — act automatically",
      "escalate_model": "Borderline confidence — re-ask a stronger model",
      "human_review": "Low confidence or irreversible — route to a person"
    }
  },
  "irreversible": {
    "type": "noul",
    "instructions": "Would acting on this answer be hard to undo (payment, email, deletion)?"
  }
}

Send a proposed answer as state and watch the gate split traffic into the execute, escalate_model, and human_review lanes.

The chain works because the gate variable is honest. Retry and cascade both act on proxies — status codes and self-reported scores — so they fire on the wrong events: retry replays a confidently wrong answer five times, cascade upgrades traffic that was already right. Jev probabilities are calibrated by RLCD training: when the gate reports 0.90, the answer is right about 90% of the time, so a threshold becomes an accuracy contract instead of a vibe. Be equally honest about the limits: the gate cannot supervise context it never sees, irreversible operations (payments, deletions, outbound email) keep a human or a deterministic rule as the final owner regardless of confidence, and calibration drifts with distribution shift — re-validate thresholds quarterly on fresh labeled samples. Keep the model as the fact-reporter; your code owns the policy.

Fallback strategy benchmark: confidence-gated chain vs retry, cascade, human-only

METRIC

Jev

Blind retry / cascade / human-only

Trigger signal

Calibrated confidence

one number per answer picks the lane

Proxies, not confidence

retry watches 5xx/timeouts, cascade watches self-reported scores, human-only waits for complaints

Added latency (P50)

~95ms at the gate

only the 0.60–0.85 middle band adds a ~2–4s stronger-model re-ask

Compounding

retry adds 2^n backoff, cascade re-generates at every hop, human queues take minutes to days

Cost shape

Input-only billing

$0.042/M at the gate; the premium model bills only for the middle band

Unbounded

retry re-bills output tokens for the same wrong answer, cascade pays premium prices per hop, human time dwarfs both

Failure mode

Fails safe

sub-threshold confidence routes to a human, never to a wrong auto-action

Loud or silent

retry repeats the identical wrong answer, cascade degrades quality silently, human-only drowns the queue

Observability

Per-decision audit trail

every hop emits answer + confidence + lane; thresholds live in code

Scattered

failures hide inside latency graphs and quality drift, with no per-decision confidence trail

Side-by-side code: blind retry vs confidence-gated chain

python / fragile default: blind retry of a single model

import time
import requests

def generate_with_retry(prompt: str, max_retries: int = 5) -> str:
    # Blind retry: the ONLY signal is transport health. A confidently
    # wrong answer passes on attempt 1 and never enters the retry path.
    for attempt in range(max_retries):
        try:
            resp = requests.post(
                "https://api.vendor-llm.example/v1/chat/completions",
                json={
                    "model": "cheap-model",
                    "messages": [{"role": "user", "content": prompt}],
                },
                timeout=30,
            )
            resp.raise_for_status()
            return resp.json()["choices"][0]["message"]["content"]
        except (requests.Timeout, requests.HTTPError):
            time.sleep(2 ** attempt)  # 1s, 2s, 4s, 8s ... up to ~31s wasted

    return ""  # silent empty-string failure after the storm

# Typical path: 1500-3000ms of generation, output tokens billed per attempt,
# and zero protection against a well-formatted wrong answer.

python / jev confidence-gated three-lane fallback chain

import requests

JEV_ENDPOINT = "https://api.typesafe.ai/v1/jev/evaluate"
AUTO_EXECUTE_CONFIDENCE = 0.85  # >= 0.85 -> act automatically

QUESTIONS = {
    "fallback_action": {
        "type": "choice",
        "instructions": "Decide the next action for this proposed answer",
        "criteria": {
            "execute": "High confidence and reversible - act automatically",
            "escalate_model": "Borderline confidence - re-ask a stronger model",
            "human_review": "Low confidence or irreversible - route to a person",
        },
    },
    "irreversible": {
        "type": "noul",
        "instructions": "Would acting on this answer be hard to undo (payment, email, deletion)?",
    },
}

def gate(state: dict) -> dict:
    # The only thing worth retrying is a transport error - and the gate
    # itself is one cheap, stateless call (~95ms, input tokens only).
    resp = requests.post(
        JEV_ENDPOINT,
        json={"state": state, "questions": QUESTIONS},
        timeout=3,
    )
    resp.raise_for_status()
    return resp.json()

def fallback_chain(task: str, answer_model, strong_model) -> dict:
    # Lane 1: the cheap model drafts; the calibrated gate decides.
    draft = answer_model(task)
    check = gate({"task": task, "proposed_answer": draft})
    decision = check["fallback_action"]

    if (
        decision["answer"] == "execute"
        and decision["confidence"] >= AUTO_EXECUTE_CONFIDENCE
        and not check["irreversible"]["answer"]
    ):
        return {"lane": "auto_execute", "answer": draft, "confidence": decision["confidence"]}

    # Lane 2: 0.60-0.85 band - one re-ask with the stronger model.
    stronger = strong_model(task)
    recheck = gate({"task": task, "proposed_answer": stronger})
    final = recheck["fallback_action"]

    if (
        final["answer"] == "execute"
        and final["confidence"] >= AUTO_EXECUTE_CONFIDENCE
    ):
        return {"lane": "escalated_model", "answer": stronger, "confidence": final["confidence"]}

    # Lane 3: sub-threshold after escalation - human queue or safe default.
    return {
        "lane": "human_review",
        "answer": stronger,
        "confidence": final["confidence"],
        "first_pass_confidence": decision["confidence"],
    }

# Typical path: ~95ms per gate call, input tokens only. Lane 2 is the
# exception, not the default - most traffic never leaves lane 1.

typescript / fragile default: blind retry

async function generateWithRetry(prompt: string, maxRetries = 5) {
  // Blind retry: only transport errors trigger the loop, so a confidently
  // wrong answer sails through untouched on the first attempt.
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const resp = await fetch("https://api.vendor-llm.example/v1/chat/completions", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          model: "cheap-model",
          messages: [{ role: "user", content: prompt }],
        }),
      });
      if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
      return (await resp.json()).choices[0].message.content as string;
    } catch {
      await new Promise((r) => setTimeout(r, 2 ** attempt * 1000));
    }
  }
  return ""; // silent failure after ~31s of backoff
}

typescript / jev confidence-gated fallback chain

const JEV_ENDPOINT = "https://api.typesafe.ai/v1/jev/evaluate";
const AUTO_EXECUTE_CONFIDENCE = 0.85; // >= 0.85 -> act automatically

const QUESTIONS = {
  fallback_action: {
    type: "choice",
    instructions: "Decide the next action for this proposed answer",
    criteria: {
      execute: "High confidence and reversible - act automatically",
      escalate_model: "Borderline confidence - re-ask a stronger model",
      human_review: "Low confidence or irreversible - route to a person",
    },
  },
  irreversible: {
    type: "noul",
    instructions: "Would acting on this answer be hard to undo (payment, email, deletion)?",
  },
} as const;

async function gate(state: Record<string, unknown>) {
  const resp = await fetch(JEV_ENDPOINT, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ state, questions: QUESTIONS }),
  });
  if (!resp.ok) throw new Error("Jev evaluate failed");
  return (await resp.json()) as {
    fallback_action: { answer: string; confidence: number };
    irreversible: { answer: boolean };
  };
}

export async function fallbackChain(
  task: string,
  answerModel: (t: string) => Promise<string>,
  strongModel: (t: string) => Promise<string>,
) {
  // Lane 1: cheap model drafts, calibrated gate decides (~95ms, input only)
  const draft = await answerModel(task);
  const first = await gate({ task, proposed_answer: draft });
  if (
    first.fallback_action.answer === "execute" &&
    first.fallback_action.confidence >= AUTO_EXECUTE_CONFIDENCE &&
    !first.irreversible.answer
  ) {
    return { lane: "auto_execute", answer: draft, confidence: first.fallback_action.confidence };
  }

  // Lane 2: 0.60-0.85 band - one re-ask with the stronger model
  const stronger = await strongModel(task);
  const recheck = await gate({ task, proposed_answer: stronger });
  if (
    recheck.fallback_action.answer === "execute" &&
    recheck.fallback_action.confidence >= AUTO_EXECUTE_CONFIDENCE
  ) {
    return { lane: "escalated_model", answer: stronger, confidence: recheck.fallback_action.confidence };
  }

  // Lane 3: sub-threshold after escalation - human queue or safe default
  return {
    lane: "human_review",
    answer: stronger,
    confidence: recheck.fallback_action.confidence,
    first_pass_confidence: first.fallback_action.confidence,
  };
}

Continue down the reliability path

Moderation with low-confidence escalationThe same 0.60–0.85 quarantine band applied to content moderation — one gate, two jobs.Model router guideChoose which model answers first and which one waits behind the escalation lane.Lead scoring recipeConfidence-gated routing in a revenue pipeline: priority ≥ 3 at 0.85 confidence auto-routes to the CRM.API referenceThe evaluate endpoint behind the gate: request shape, response fields, limits.Test thresholds in the PlaygroundRun real states through the gate and watch the confidence split.