Benchmarks

Method

• Pre-registered 3aba674 · 25e3c87

  Pre-registered by commits in Courant’s development repository, which is not public: the judges before any request (3aba674, 15 Oct 2026), the run at scale before any run (25e3c87, 10 Oct 2026; its versions 194fa76, 13 Oct 2026).

• The runs 140 right · 140 wrong problem · 140 solver fault

  Every judge got the same 420 runs and the same task: 140 right, 140 the wrong problem, 140 with a solver fault; Courant’s verdicts are the ones it recorded on them at scale.

  The judges saw a rebuilt case configuration, not the case Courant read.

• Courant’s gate version 2 · courant 0.5.0 runs version 1

  Courant’s verdicts here are gate version 2’s; courant 0.5.0, the release, judges without what version 2 adds: the cell aspect ratio computed from the mesh files when a case has no checkMesh log and each field’s residual bar from the case’s own residualControl, with a stalled-residual check. Version 1’s interim, stopped at 273 of its 2,100 runs, accepted 6 of 91 faulted runs given the task (95% CI 3.06–13.65%), inconclusive. Versions are never pooled.

• Scope and spend $77.50 · 1,262 model requests

  Tutorial-sized OpenFOAM v2406 runs of seven families; no customer data. $77.50 in model requests, 1,262 of them.

Questions

• How do I verify an OpenFOAM simulation?

  Install Courant (pip install 'courant-dev[mcp]') and run courant verify ./my_case on the case directory. It reads the solver log and the case files and returns accept, reject or insufficient, each reason citing the file it read, the method, the threshold and the value. Give it the task’s specification and it also checks that the run is the problem that was asked.

• How do three Claude models compare with Courant at judging OpenFOAM runs?

  Given the same 420 OpenFOAM tutorial runs and the task, Claude Opus 5.5 accepted 69 of 264 wrong runs, Claude Sonnet 5.5 76 of 263 and Claude Haiku 5.5 52 of 249; Courant, at gate version 2, accepted 0 of 264. Of the right runs, Courant rejected 10 of 140, the models 5 of 140, 0 of 138 and 17 of 122, in the same order. A statement about these models, this prompt and these dates, on tutorial runs, never about language models in general. The judges saw a rebuilt case configuration, not the case Courant read.

• What does Courant check, and what does it not?

  It reads what a finished run left behind, the solver log and the case files, and checks convergence, stability, bounded fields and the setup. Given the task, it also checks that the case is the problem that was asked: on the same runs it accepted 118 of 140 wrong-problem runs without the task and 0 of 140 with it. It runs no solver, changes no case and does not replace a validation experiment.

• Does it send my files anywhere?

  Courant runs on your machine and has no account, no telemetry and no service of its own; it never sends your runs, cases or results anywhere. Its checks open no network connection. Only these contact the network, only when you run them, and each names the host first: courant update downloads a release from github.com; courant doctor --models reads OpenRouter’s public model list; and a language model you choose (an llm: judge for the Courant Test, or courant spec extract) sends that task’s text to the model provider with your own API key.

• How is its error rate measured?

  On real OpenFOAM tutorial runs whose truth is known by construction: right runs, runs with an injected solver fault, and the right physics on the wrong problem. At scale, given the task, it accepted 0 of 609 faulted and 0 of 609 wrong-problem runs, and rejected 34 of 609 right runs, a rate that missed its pre-registered bar; without the task it accepted 3 of 609 faulted runs, which also missed its bar. Both studies on /benchmarks were pre-registered before their first request or run and published whatever they showed, every rate with its denominator and its 95% CI. Courant’s counts here are gate version 2’s; courant 0.5.0, the release, judges without the changes version 2 adds, and version 1’s interim accepted 6 of 91 faulted runs given the task (95% CI 3.06–13.65%), inconclusive. Versions are never pooled.

Claude and the Claude logo are trademarks of Anthropic, PBC.