Claude Opus 5.5 and Sonnet 5.5 as typed judges
The same 1,834 questions on the same 449 recorded states as the typed-judges benchmark, byte for byte, asked of the two Claude 5.5 models.
Results
Graded questions are the 726 whose answer the case kind fixes: the defect question, the verdict, each per-test label, and for plan review whether any lens blocks. A positive case has no defect; a negative or pressure case has one. An answer is right when its probability is on the correct side of 0.5.
| System | Model | Run | Right | Wrong of 726 | Brier | API-equivalent per 1,000 exchanges |
|---|---|---|---|---|---|---|
| Opus 5.5 | claude-opus-5-5 | 2026-10-06 | 94.5% | 40 | 0.051 | $326 |
| Opus 5 | claude-opus-5 | 2026-09-25 | 91.5% | 62 | 0.066 | $433 |
| Sonnet 5.5 | claude-sonnet-5-5 | 2026-10-06 | 85.8% | 103 | 0.102 | $164 |
| Jev | jev (TypeSafe, hosted) | 2026-09-24 | 85.4% | 106 | 0.107 | not comparable |
| Laya | laya | 2026-09-24 | 63.4% | 266 | 0.249 | local |
Laya's wrong count is computed from its 63.4% and shown for reference. Jev was priced by its own API, so its cost is not on the same scale.
Paired tests
Every system answered the identical questions, so the fair comparison is per question. The p-value is an exact two-sided McNemar test on the discordant pairs.
| Comparison | A right, B wrong | B right, A wrong | Both wrong | p |
|---|---|---|---|---|
| Opus 5.5 vs Opus 5 | 31 | 9 | 31 | 0.0007 |
| Sonnet 5.5 vs Jev | 54 | 51 | 52 | 0.85 |
| Sonnet 5.5 vs Opus 5.5 | 8 | 71 | 32 | <0.0001 |
| Opus 5.5 vs Jev | 84 | 18 | 22 | <0.0001 |
Opus 5.5 is a real improvement on Opus 5. Sonnet 5.5 and Jev are statistically level, and they miss different questions.
By reviewer role
| Role | Opus 5 | Opus 5.5 | Sonnet 5.5 | Jev |
|---|---|---|---|---|
| code review | 92.2% | 93.3% | 93.3% | 88.3% |
| plan review | 75.0% | 73.3% | 73.3% | 75.0% |
| spec review | 87.6% | 97.2% | 76.4% | 75.8% |
| task review | 97.2% | 97.8% | 93.9% | 92.2% |
| test review | 95.3% | 97.7% | 82.8% | 89.8% |
The cheaper model is not a uniformly smaller copy. Sonnet 5.5 matches Opus 5.5 on code review and falls furthest on spec review and per-test labels.
Cost
All three Claude runs used the claude.ai subscription through claude -p, so nothing was charged. The figures are the API-equivalent price claude -p reports per call: Opus 5 $194.26, Opus 5.5 $146.57 and Sonnet 5.5 $73.52 for the 449 exchanges. Opus 5.5 came out 24.5% cheaper than Opus 5 on this workload; Anthropic states 40% on typical workloads. Sonnet 5.5 cost 50.2% of Opus 5.5. Median latency per exchange: 19.7 s, 18.5 s and 17.5 s.
Caveats
One run per model, no repeat variance. Hand-written corpus cases, not live work. Each judge answers typed questions without tools, not the production judge that reads files and writes findings. The prompt, schema and four-way concurrency are identical to the 2026-09-24 Opus 5 run, reconstructed from that session and checked by reproducing its published scores exactly (91.5%, 85.4%, 63.4%).
Reproduce
Code: replay_claude.py (the replay), score_three.py (grading), paired.py (paired tests). Data: scores.json, paired.json, runs.json. Run from the forge checkout with FORGE_ROOT set: python3 replay_claude.py claude-opus-5-5 opus-5-5, then python3 score_three.py jev claude opus-5-5.