testing · fixtures · CI gates · local engineering pattern
A practical guide to Jev regression testing: freeze replay fixtures, assert schemas and decisions, compare probabilities with justified tolerances, gate CI on meaningful diffs, diagnose drift, and keep a human reviewer in charge.
A useful Jev regression test is a versioned measurement instrument, not a saved prompt. Replay the same redacted state, question, criteria, provider path and resolved model; assert the typed decision and required side effects exactly; treat probabilities as bands or deltas; and fail CI only on a policy you wrote before seeing the candidate result. A red build is a request for diagnosis, not permission to keep calling until the preferred answer appears.
This is an application-owned testing pattern, not a provider guarantee. Freeze the complete evaluator artifact, keep raw responses and failures, choose tolerances from labeled data and repeated runs, and review every baseline change. No monthly search-volume estimate is used here.
Regression testing becomes trustworthy when every promise has an owner.
local fixture contract
Store the state projection, question text, criteria, provider path, resolved model identifier and decision rule together. A changed alias or question is a changed measuring instrument, even when application code did not move.
local release policy
Require the labeled decision and the required side effects exactly. Compare probabilities with a declared tolerance or band; never fail a release because a floating-point value moved from 0.873 to 0.871.
human decision
A red diff is evidence, not a verdict about safety. Preserve the run, inspect the affected slice, and require a named reviewer before accepting a new baseline or widening a tolerance.
The fixture is the smallest complete record that lets another engineer reproduce the decision and explain a diff.
fixture_idStable ID; never use a row number as identity.
stateRedacted evidence with unstable timestamps and secrets removed.
questionsExact type, instructions and criteria used in the run.
expectedHuman label plus local pass/review policy, not a provider promise.
instrumentProjection, question, provider and resolved model versions.
The JSON below is an illustrative manifest, not a Jev configuration format. Keep the raw provider response, request metadata and adapter hash beside it. If a model alias resolves differently, record that as an instrument change instead of silently rewriting the baseline.
{
"suite": "refund_completion_v3",
"fixture_id": "completed-017",
"input": {
"state": "Order ORD-1042 was delivered. The refund was issued in full.",
"questions": {
"completed": {
"type": "noul",
"instructions": "Decide whether the refund process is complete.",
"criteria": {
"true": "The evidence says the refund was issued in full.",
"false": "The refund is pending, partial, or not evidenced."
}
}
}
},
"expected": {
"decision": "true",
"pass_min": 0.82,
"review_min": 0.55
},
"instrument": {
"provider": "your-provider",
"model": "pinned-model-id",
"projection_version": "refund-trace-v2",
"question_version": "completion-v3"
}
}Typed output narrows the assertion surface; it does not remove the need for ordinary test code.
| Signal | What to assert | Recommended gate |
|---|---|---|
Noul / boolean | The labeled decision is true or false, and the probability stays in the declared pass/review band. | Exact decision; band or tolerance for probability. |
Choice | The selected option is the expected label. Inspect the top-two margin when a near tie is risky. | Exact label; margin threshold only when justified by the task. |
Score / rubric | The score is inside an accepted range, while the raw score remains available for trend analysis. | Range or aggregate delta; do not overfit one example. |
Application state | The workflow created the required record, tool call, queue or status. A convincing sentence is not proof that the side effect happened. | Exact database/API/UI assertion owned by ordinary test code. |
Always fail closed on an invalid response. A timeout, malformed distribution or missing field is not the same thing as a negative decision. Preserve the distinction in the report so infrastructure failures do not pollute semantic metrics.
const PROBABILITY_TOLERANCE = 0.05;
function checkCase(baseline, current, expected) {
const decisionChanged = baseline.decision !== current.decision;
const probabilityDelta = Math.abs(
baseline.probability - current.probability,
);
if (current.decision !== expected.decision) {
return { status: 'fail', reason: 'decision changed against the label' };
}
if (decisionChanged && probabilityDelta > PROBABILITY_TOLERANCE) {
return { status: 'fail', reason: 'decision flip exceeded tolerance' };
}
if (probabilityDelta > PROBABILITY_TOLERANCE) {
return { status: 'review', reason: 'probability moved beyond tolerance' };
}
return { status: 'pass', probabilityDelta };
}Use the smallest policy that matches the risk. The numbers below are examples, not universal defaults.
For a labeled fixture, the selected choice or boolean result is usually the hard assertion. A probability moving from 0.873 to 0.871 is not a regression by itself; a changed decision is a different signal.
For thresholded automation, store pass, review and fail bands. Fit the boundary on held-out labels or repeated runs, then keep the formula and denominator beside the gate. Do not turn one attractive run into a guarantee.
A case near the threshold deserves review or a repeated run, not an automatically widened tolerance. Track flip rate, top-two margin and slice impact when the decision is consequential.
Compare means, error rates or calibration measures only with their sample size and slice definition. Resample at the independent unit, such as a session or customer, when repeated events are correlated.
Run a small deterministic contract suite on every change, then spend live-provider budget on a bounded smoke or release suite.
name: Jev regression suite
on:
pull_request:
push:
branches: [main]
jobs:
jev-regression:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- run: pnpm install --frozen-lockfile
- run: pnpm eval:jev -- --fixtures evals/fixtures.jsonl --baseline evals/baseline.json --tolerance 0.05 --write-current artifacts/jev-current.json --markdown artifacts/jev-regression.md
- uses: actions/upload-artifact@v4
if: always()
with:
name: jev-regression-report
path: artifacts/
The workflow uses `eval:jev` as a placeholder for your repository-owned wrapper. A community harness may offer similar baseline and tolerance flags, but neither that CLI nor this page is an official provider interface. Make the wrapper exit non-zero only for pre-declared gates, and upload the full report on both pass and fail.
A useful failure report makes the next action obvious and prevents “retry until green.”
| Failure class | Evidence to retain | Next action |
|---|---|---|
| Malformed request or parse error | Payload hash, schema error, response body and adapter version. | Fix the contract or reject the provider response before scoring it. |
| Timeout, rate limit or auth failure | HTTP status, request ID, attempt count, backoff and provider/model identity. | Classify as infrastructure first; do not convert no-result into a false decision. |
| Same instrument, labeled decision changed | Paired old/current output, probability delta, fixture tags and affected slice. | Treat it as a semantic regression until a reviewer explains the change. |
| Fixture or label defect | Redaction diff, source reference, label provenance and reviewer history. | Correct it with provenance and keep the old record; never delete history to make CI green. |
Print old/current distributions, model and provider identity, fixture tags, request IDs, latency and retry metadata. Keep no-result cases separate from false decisions, and keep a baseline update as a reviewed change with a reason.
Automate collection and comparison; reserve the judgment call for the people who own the risk.
Confirm the fixture is still representative and redacted correctly. If the product behavior changed intentionally, update the requirement and its provenance rather than only changing the expected label.
Inspect the paired evidence, not just the aggregate score. Read the state projection, question, criteria and raw response for every critical failure.
Check whether the change is isolated or concentrated in a slice: language, source, severity, evidence availability, option count or boundary distance.
Choose one outcome: fix the adapter, retry eligible infrastructure failures, revise the fixture with provenance, block the release, or accept a new baseline with a named reviewer and expiry date.
Keep the old and new artifacts. A baseline is a decision record, not a disposable cache.
Do not make a probability threshold sound like a safety certificate. A local gate says how this repository handles this fixture set under this instrument; it does not prove provider accuracy, future stability or business safety outside the measured scope.
At minimum: a stable fixture ID, redacted state, the exact typed questions and criteria, a human label or accepted outcome, the local decision policy, and the provider/model/instrument versions. Keep the raw response and request metadata beside the fixture so a changed result can be diagnosed instead of merely marked red.
Usually no. Assert the typed decision and required application side effects exactly, then compare probability movement with a declared tolerance or band. Exact floating-point snapshots are brittle. Choose the tolerance from held-out labels, repeated-run variation and the cost of a wrong decision; the example 0.05 in this guide is not a universal default.
Run a provider-free contract suite on every pull request, then run a bounded live-provider smoke or release suite with pinned fixtures. Save the current run, compare it with the reviewed baseline, fail only on pre-declared critical cases or aggregate gates, and upload the report even when the job fails. Keep the CLI wrapper in your repository so the policy is reviewable.
It may be a semantic regression, an intentional requirement change, a changed question or state projection, a provider/model change, a fixture defect, or an infrastructure error misclassified as a decision. Compare the complete instrument and raw response first. Do not update the baseline until a reviewer can name the cause and the intended action.
No. It can provide repeatable evidence for the fixtures, labels, instrument and gates you actually measured. It cannot establish a provider guarantee or cover unrepresented traffic. Add critical-case hard gates, slice analysis, delayed human labels and production monitoring, and state the scope of every claim.
Review critical-case failures, threshold-near cases, new or removed fixtures, concentrated slice regressions, baseline changes and any result caused by a provider or model version change. Human review is especially important when the aggregate metric looks stable but a high-consequence fixture flips.