Evaluation & Observability

Jev scores correctness, complexity, tests and security; code ranks areas to improve.

↳ Compares scores across checkpoints while leaving code changes to the primary Agent.

Runs the same labeled text tasks through both backends and records probabilities, latency and failures.

↳ Helps examine task-specific accuracy and whether confidence scores support chosen thresholds.

Experiments send action candidates or typed questions to Jev, then execute or record the answers.

↳ Includes source, experiment notes and some offline replays for comparing decision designs.

Choice selects the next word; Noul evaluates candidate chunks and stopping conditions.

↳ Exposes the limitations of using a decision model as a text generator.

Compares stepwise Choice decoding with selection from complete candidate replies.

↳ Provides decoder methods, experiment traces and documented failure cases.

Builds candidate sets from traces and submits three choice questions.

↳ Provides evaluation scripts and author results; some baselines generate answers while Jev selects candidates.

Sends selected records to Jev for judgments about execution claims and evidence sufficiency.

↳ Only reads and sends records on explicit invocation; judgments do not replace real tests.

Ranks candidate passages with Choice, Noul and rubric scores, then computes retrieval metrics.

↳ Publishes raw responses, scoring code and per-dataset results for inspection.

Chooses moves from direction candidates, then checks collisions, paths and completion.

↳ Keeps experiment settings and failures visible for studying limitations.

Jev assesses progress and completion signals; deterministic acceptance checks still decide whether the work passes.

↳ Separates a model’s assessment from executable completion checks.

Judges which checks apply and can add checks when thresholds and rate limits permit.

↳ Records models, thresholds, latency and selection reasons alongside the baseline.

Turns conditions into yes/no probability questions and returns local-threshold verdicts plus probabilities.

↳ Adds structured checks to task outcomes without making Jev the controller of the whole messaging system.

Jev chooses from locally generated move candidates while the experiment varies board and tactical information.

↳ Supports labeled replays and bring-your-own-key live games with per-move records.

Runs fixed questions and measures accuracy, calibration, coverage and escalation rates.

↳ Connects threshold selection and model-change checks to reports and CI.

Selects legal chess moves or judges whether an utterance addresses each NPC.

↳ Publishes labeled data, raw requests and responses, and evaluation code.

Jev selects observed controls and operations; test code owns expected values and pass criteria.

↳ Records exploratory behavior separately from contract acceptance.

Local checks handle machine-verifiable requirements; Jev assesses the remaining semantic conditions.

↳ Connects goals with checkable criteria without replacing actual acceptance evidence with model opinions.

Asks Choice and Noul questions about eligibility, then combines them into include or exclude decisions.

↳ Records metrics for specific dataset slices and question designs to examine screening errors.

Calls Jev on defined questions and labeled cases, then analyzes errors, calibration, and difficulty effects.

↳ Links research claims to experiment code, data, and an evidence ledger.

Jev checks the diff, selects evidence from actual hunks and adjudicates candidate findings.

↳ The caller supplies diffs and task boundaries; the engine does not scan the repository itself.

Asks Jev about file-quality properties; code composes scores and ordering.

↳ Breaks scores into named properties and caches results by content.

Jev classifies change evidence against configured rules; code maps answers to review outcomes.

↳ Feeds semantic-rule findings to coding Agents without replacing type checks, tests or security audits.

Judges support messages or chooses lanes and target speed from structured simulated sensors.

↳ Shows inputs, probabilities and resulting behavior together.

Asks whether predefined traits are present or how strongly they appear, recording scores, latency and Token usage.

↳ Compares rating scales, input variants and batching over a frozen sample.

Collects Noul and Choice outputs and compares them with labels for error, accuracy and stability.

↳ Keeps per-call records and experiment notes to qualify conclusions.

Maps the same tasks to Jev Choice/Noul questions and normalizes answers to a shared result format.

↳ Preserves comparison methods and results for inspecting model differences.

Jev can identify completion claims and semantic-rule issues; stop gates depend on ledger facts and local rules.

↳ Separates execution evidence from model opinion rather than letting Jev alone certify completion.

Sends controlled variants of fixed tasks and records choices, probabilities and raw request-response evidence.

↳ Lets readers inspect individual cases rather than infer broad capability from simple-task success.

Collects judgments on matched tasks and computes confidence intervals and repeat-input stability.

↳ Publishes data processing, runner and statistics code with model-specific results.