Courant
Install the gate

1,827 real runs, and the two bars we missed

Of six hypotheses pre-registered on 1,827 real OpenFOAM runs, four held and two were refuted.

Courant-10K exists to test Courant on real solver runs in the thousands, with every bar fixed before the first run.

How it was built

Each draw from seven OpenFOAM tutorial families was run three ways.

Once right. Once with a catalogue fault. Once with the right physics on the wrong problem.

Truth is by construction. Each hypothesis is decided on the Wilson upper bound of its interval, against a bar fixed before any run. Exclusions were fixed in advance too.

  1. Faulted runs accepted, without the task, bar 1%, 3 of 609, 95% CI 0.17–1.44%, Refuted
  2. Faulted runs accepted, with the task, bar 1%, 0 of 609, 95% CI 0.00–0.63%, Supported
  3. Wrong-problem runs accepted, with the task, bar 1%, 0 of 609, 95% CI 0.00–0.63%, Supported
  4. Right runs rejected, with the task, bar 2%, 34 of 609, 95% CI 4.02–7.70%, Refuted
  5. Right runs left insufficient, with the task, bar 5%, 0 of 609, 95% CI 0.00–0.63%, Supported
  6. Verifier exceptions, 0 exceptions, 0 in 3,654, Supported
Courant-10K: the gate on thousands of real OpenFOAM runs · pre-registered 10 Oct 2026 · A hypothesis is supported only when its whole 95% CI sits under a bar fixed before any run. Every number

What held

With the task’s specification, on 1,827 runs:

  • Faulted runs accepted: 0 of 609.
  • Wrong-problem runs accepted: 0 of 609.
  • Right runs left insufficient: 0 of 609.
  • Verifier exceptions: 0 in 3,654 verifications.

At 0 of 609, its 95% CI 0.00–0.63% sits wholly under its bar of 1%.

What missed

Both misses are published as refuted.

  • 3 of 609 faulted runs accepted without the task, against a bar of 1% (95% CI 0.17–1.44%). Without the task, 3 hotRoom runs set to laminar on a turbulent case were accepted; with it, none was.
  • 34 of 609 right runs rejected with the task, against a bar of 2% (95% CI 4.02–7.70%). 30 of 87 pitzDaily right runs (epsilon still being clipped late in the solve) and 4 of 87 hotRoom (Courant number above the stability limit over the final fifth of the run, adjustTimeStep off).

Which gate

Courant’s counts here are gate version 2’s; courant 0.6.0, the release, gives courant 0.5.0’s verdicts by default on 1,827 of 1,827 Courant-10K runs, with version 2’s changes behind --checks next, and version 1’s interim accepted 6 of 91 faulted runs given the task (95% CI 3.06–13.65%), inconclusive. Versions are never pooled.

Method and every number: Benchmarks and RESULTS.md.