Given the task, three Claude models let wrong runs through
Given the same 420 real OpenFOAM runs and the task, Courant accepted 0 of 264 wrong runs. Claude Opus, Sonnet and Haiku 5.5 accepted 69 of 264, 76 of 263 and 52 of 249.
An AI agent that runs a simulation needs a judge before it reports. The obvious judge is another language model.
We measured how often that judge is wrong.
The test
We took 420 real OpenFOAM v2406 runs from Courant-10K: 140 right and 280 wrong. Truth was fixed by construction, before any judge saw a run.
Every judge got the same files: the solver log, the case configuration, the mesh metrics and the task’s specification.
Each was asked one question: is this run a valid result of the task?
The study was pre-registered on 15 Oct 2026, before the first request. Each model answered once. We published what it showed.
What got through
- Courant,0 of 264 wrong runs let through,10 of 140 right runs rejected
Claude Opus 5.5,69 of 264 wrong runs let through,5 of 140 right runs rejected- Claude Sonnet 5.5,76 of 263 wrong runs let through,0 of 138 right runs rejected
- Claude Haiku 5.5,52 of 249 wrong runs let through,17 of 122 right runs rejected
Abstentions are left out of each denominator. For each model, an exact one-sided McNemar test against Courant gave p of 2.22e-16 or less, against α 0.0167.
Where the models miss
Most misses were runs that solved the wrong problem: the right physics, a wrong input, a clean convergence.
Claude Opus, Sonnet and Haiku 5.5 accepted 61 of 140, 59 of 139 and 46 of 129 such runs.
The specification was in the prompt. Comparing a value the task names with the value the solver read is mechanical.
It is also where the models failed most.
Where Courant does worse
Courant rejected 10 of 140 right runs: more than Claude Opus 5.5 (5 of 140) and Claude Sonnet 5.5 (0 of 138), fewer than Claude Haiku 5.5 (17 of 122).
Courant’s rejections are 8 of 20 pitzDaily and 2 of 20 hotRoom runs.
What this does not show
These are these models, this prompt and these dates. The runs are tutorial-sized. The judges saw a rebuilt case configuration; Courant read the case.
Method and every number: Benchmarks and RESULTS.md.