Typed decision models as code-review judges

Jev and every open-source Jev-class alternative, asked the same typed questions about the same 150 review cases, scored against verdicts fixed before any of them ran.

Ten systems · five reviewer roles · 1,834 questions · Claude as the reference · 2026-09-25, CLM-8B added 2026-09-27

Two things came out of it that matter more than the leaderboard.

What was measured

A code-review pipeline has five reviewer roles, each answering a bounded question about a piece of state. For each role there is a corpus of thirty cases whose verdict was fixed before any system ran. Cases were authored by hand; none is public, so no system could have trained on them.

Each corpus has three kinds of case, ten of each. A positive is clean; any blocking or major finding is a false fail. A negative plants a defect at a named path. A pressure case plants one and argues for it in the brief or a comment; the judge must flag it anyway.

How the questions were chosen

Jev-class systems take a state and typed questions, a yes/no with a probability, a choice over named options with a distribution, an ordered score, and return no text. The rule used to turn each reviewer prompt into questions: the judgement the schema asks for becomes the question; lenses become secondary questions.

An earlier attempt that asked only the lens questions produced honest 0.5s and never asked for the judgement. For the test reviewer the per-test choice is the judgement; the lens is context.

How the state was built

Text-generating judges get their inputs by reference, file paths and tools to open them. A typed model has no tools, so the state must carry the content. Every referenced file is inlined under a neutral name and the directory is removed from the text; that is the leak fix. The file a test id names is inlined too. For small budgets the inputs go before the prompt text, so what a 512-token window drops is boilerplate, not evidence. Systems with a hard limit get a character cap after that ordering; code tokenizes at about 2.3 characters per token.

Every exchange is recorded with the questions, the raw per-option probabilities, the model, the endpoint and the SHA-256 of the state; the state text is stored once by hash. That is what allowed the replay below: identical bytes to every system.

How answers were graded

A positive passes with no finding at or above major. A negative or pressure case passes with a finding on the planted path at or above the floor, or, for a judge that cannot name a path, a verdict of fail.

The systems

Test reviewer, the role every system ran

Accuracy is of the per-test probability against the known kind; 0.67 is what a constant "defect" answer earns on a 20 to 10 split. Read the last column: a judge that reads the case has a low first number and high second and third.

The other four roles

Laya's 28 of 30 on code review against Jev's 26 is the flag-everything route the diff-role grader still allows: its probability of a defect is 0.73 on clean diffs and 0.70 on defective ones. Kev 4B on the diff roles puts 0.03 to 0.15 on everything, clean or not. Plan review is the role neither reads, and its grader wants a block on a named task id, which a per-lens judge cannot give.

Head to head on identical questions: Jev and Laya

All 449 recorded exchanges were replayed through both in one sitting, 1,834 questions each, byte-identical state and questions.

The 3,306 recorded questions, replayed through the two kept models

Nine systems answered 3,306 questions in total, most of them the same question about the same case asked of different systems. Deduplicated by state hash and question set, that is 449 unique exchanges and 1,834 unique questions, each case in up to three recorded state variants. Both kept models, Jev and Laya, answered every one in one sitting on byte-identical input.

All 1,834 questions again, with Claude as the reference

The replay above shows Jev and Laya disagree; it does not say who is right. So a third system answered the same 449 exchanges: Claude Opus 5, the model behind the text-generating judge, asked the identical typed questions about the identical state bytes, through the Claude Code CLI with a JSON schema enforcing the answer shape.

Of the 1,834 questions, 726 have a known right answer, because the case kind fixes it: the 269 defect yes/no questions, the 269 verdicts, the 128 per-test labels, and for plan review whether any lens says block, 60 in all. Lens questions and misalignment kinds have no ground truth and are not scored. Accuracy is the probability against 0.5; Brier is the squared error of the probability, lower is better.

Probability of a defect, clean → defective

A system that reads the case has a low first number and a high second.

Who got each question right

The same questions, put to CLM-8B

Four days after this report, Stanford and NVIDIA released CLM-v0.1-8B: a frozen Qwen3-8B encoder with two contrastive projection heads of about 20M parameters, scoring a state against candidate actions with no text generation, Apache 2.0. It is the first open model pitched directly at Jev's job, so on 2026-09-27 it was put on the same questions.

Six plan-review positives were rewritten after the Claude replay, so 443 of the 449 exchanges are reproducible today and 720 questions are gradable, not 726. Jev was re-run on the same 443 on 2026-09-26, which is why its figures here differ by a point from the section above: a fresh run on a slightly different corpus, not a re-grade.

Probability of a defect, clean → defective

CLM-8B's two numbers are within 0.02 of each other in every role, and on test review it returns 1.0 for every per-test label.

Why a typed judge at all

A text-generating judge costs about $0.57 and 45 seconds per case. Re-running a thirty-case corpus after a prompt edit is $17 and 25 minutes; recalibrating five roles is about $90 and two and a half hours. Jev answers a corpus for half a cent in twelve seconds, the local models for nothing in a similar time. That is what makes prompt iteration on judges affordable, whichever typed model is chosen.

Findings

1. Check what the state leaks before believing a score. Paths, IDs and filenames are text to a model. The first 30 of 30 was the label.
2. A score a constant answer can earn is not a score. Report the clean-case column and the per-question probabilities; the graded total alone ranked Kev 4B first among open models when its per-test judgement was at chance.
3. Sub-1B encoders do not discriminate on review-sized states. Laya, Von and Kev 0.8B give the same probability whatever the case. Shrinking the state to the test file and the brief did not help them.
4. Nimble 9B reads the test but is starved. Its 2,048-token limit removes the brief and criteria on every role but the minimal one; it is the only open model with a calibrated per-test judgement.
5. Kev 4B is confident and wrong on diffs. Decisive probabilities, near-zero probability of a defect on planted defects in the alignment and code roles.
6. Jev is the reference among typed models. Accuracy 0.89 to 0.97 with Brier 0.05 to 0.11 on four roles, and the only model whose lens answers stay low on clean tests.
7. Claude is the most accurate; Jev is the best value. On all 1,834 replayed questions, Claude is right on 91.5% of the 726 gradable ones, Jev on 85.4%, Laya on 63.4%. Jev costs about one three-thousandth as much and answers fifty times faster.
8. CLM-8B, the open model built for this job, does not do it out of the box. On the same 720 gradable questions it is right on 56.1% with a Brier of 0.283, and its probability of a defect does not move between clean and defective states. Jev is right on 86.5%.
9. Two judges are worth their cost only if their errors are independent. No case was missed by both Jev and the text-generating judge; that is a hint, and the real measurement is the disagreement rate on live runs, which this benchmark does not contain.

Limitations: thirty cases per role, hand-authored; run-to-run variance of about one case in thirty for the hosted model; the diff-role grader still admits the flag-everything route; local latencies were measured with other models loaded; Nimble and Kev 4B ran on a shared free-tier GPU with capped or reordered state; NanoJev was not run; CLM-8B ran as an 8-bit quantization of its encoder on a laptop, past its published 2,048-token window, so its latency is the laptop's.