The Courant Test.
Thirty-one real runs. Labels locked.
| Case | Solver | Runs | Faulted | Healthy | Split |
|---|---|---|---|---|---|
| Lid-driven cavity | icoFoam | 7 | 6 | 1 | held in |
| Backward-facing step | simpleFoam | 7 | 6 | 1 | held in |
| Turbulent cavity | pisoFoam | 5 | 4 | 1 | held in |
| Dam break | interFoam | 3 | 1 | 2 | held in |
| Planar contraction | pimpleFoam | 3 | 2 | 1 | held in |
| Compressible cavity | rhoPimpleFoam | 3 | 2 | 1 | held in |
| Heated room | buoyantPimpleFoam | 3 | 2 | 1 | held out |
OpenFOAM v2406 tutorials, run in Docker, faults injected by editing the case the way an engineer’s mistake would. The labels, the scorer and the split are hash-locked; the scorer refuses to score when a lock changes without a recorded relabel decision.
certification/RESULTS.json 6f67ffab5cbed10d…
courant/ctest/data/judge-v0.labels.jsonl 207c35b8350ae1fa…
courant/ctest/score.py 452871b5230376ed…
courant/ctest/splits.py 2eb270d55e1e9db5…
Score any judge.
$ courant test run --judge courant all tasks 31 · accuracy 31/31 · false accepts 0/23 · false alarms 0/8 · abstain 0/31 · top-2 23/23 held_in tasks 28 · accuracy 28/28 · false accepts 0/21 · false alarms 0/7 · abstain 0/28 · top-2 21/21 held_out tasks 3 · accuracy 3/3 · false accepts 0/2 · false alarms 0/1 · abstain 0/3 · top-2 2/2 $ courant test run --judge constant:TRUSTWORTHY all tasks 31 · accuracy 8/31 · false accepts 23/23 · false alarms 0/8 · abstain 0/31 · top-2 0/23 held_in tasks 28 · accuracy 7/28 · false accepts 21/21 · false alarms 0/7 · abstain 0/28 · top-2 0/21 held_out tasks 3 · accuracy 1/3 · false accepts 2/2 · false alarms 0/1 · abstain 0/3 · top-2 0/2 $ courant test run --judge constant:NOT_TRUSTWORTHY all tasks 31 · accuracy 23/31 · false accepts 0/23 · false alarms 8/8 · abstain 0/31 · top-2 0/23 held_in tasks 28 · accuracy 21/28 · false accepts 0/21 · false alarms 7/7 · abstain 0/28 · top-2 0/21 held_out tasks 3 · accuracy 2/3 · false accepts 0/2 · false alarms 1/1 · abstain 0/3 · top-2 0/2 $ courant test run --judge llm:<provider>:<model> a model, scored the same way
A judge that trusts every run is right on 8 of 31 and accepts every wrong result. One that trusts nothing accepts no wrong result and rejects every good one. Courant: 31 of 31 correct, 0 of 23 wrong results accepted (95% CI 0–14.3%), 0 of 8 good results rejected (95% CI 0–32.4%). Held out of development: 3 of 3 correct.
Every run, every verdict.
| Run | Truth | Verdict | Rule | Leading cause | Why |
|---|---|---|---|---|---|
| Heated room healthy | healthy | Accept | VER-5.6 | NUM.MISSING_ADJUSTABLE_DT | No check failed. |
| Heated room cfl blowup | faulted | Reject | VER-5.2 | NUM.MISSING_ADJUSTABLE_DT | run.crash · value SIGFPE |
| Heated room laminar on turbulent | faulted | Reject | VER-5.4 | PHYS.LAMINAR_TURBULENT | physics.laminar_unstable |
| Lid-driven cavity healthy | healthy | Accept | VER-5.6 | — | No check failed. |
| Lid-driven cavity bad initial field | faulted | Reject | VER-5.2 | NUM.CFL_VIOLATION | run.crash · value SIGFPE |
| Lid-driven cavity cfl blowup | faulted | Reject | VER-5.3 | NUM.CFL_VIOLATION | cfl.max · value 9.66086 · threshold 1.0 |
| Lid-driven cavity extreme aspect ratio | faulted | Reject | VER-5.2 | NUM.CFL_VIOLATION | run.crash · value SIGFPE |
| Lid-driven cavity high skewness | faulted | Reject | VER-5.2 | NUM.CFL_VIOLATION | run.crash · value SIGFPE |
| Lid-driven cavity missing dict | faulted | Reject | VER-5.2 | ENV.MISSING_DICT | run.crash · value FOAM FATAL ERROR |
| Lid-driven cavity transport units | faulted | Reject | VER-5.4 | PHYS.TRANSPORT_UNITS | physics.nu · value 1e-09 |
| Dam break cfl blowup | healthy, relabelled | Accept | VER-5.6 | NUM.CFL_VIOLATION | No check failed. |
| Dam break healthy | healthy | Accept | VER-5.6 | PHYS.LAMINAR_TURBULENT | No check failed. |
| Dam break fixed dt on adaptive solver | faulted | Reject | VER-5.3 | NUM.MISSING_ADJUSTABLE_DT | cfl.max · value 7.37424 · threshold 1.0 |
| Planar contraction healthy | healthy | Accept | VER-5.6 | PHYS.LAMINAR_TURBULENT | No check failed. |
| Planar contraction cfl blowup | faulted | Reject | VER-5.3 | NUM.CFL_VIOLATION | convergence.residual · value 0.015446261 |
| Planar contraction fixed dt on adaptive solver | faulted | Reject | VER-5.3 | NUM.MISSING_ADJUSTABLE_DT | convergence.residual · value 0.015446261 |
| Turbulent cavity healthy | healthy | Accept | VER-5.6 | — | No check failed. |
| Turbulent cavity cfl blowup | faulted | Reject | VER-5.3 | NUM.CFL_VIOLATION | convergence.residual · value 0.070853 |
| Turbulent cavity laminar on turbulent | faulted | Reject | VER-5.2 | PHYS.LAMINAR_TURBULENT | run.crash · value SIGFPE |
| Turbulent cavity loose solver tolerance | faulted | Reject | VER-5.4 | NUM.SOLVER_TOLERANCE | solver.loose_tolerance · value 0.1 · threshold 0.0001 |
| Turbulent cavity wrong turbulence model | faulted | Reject | VER-5.2 | ENV.MISSING_DICT | run.crash · value FOAM FATAL ERROR |
| Compressible cavity healthy | healthy | Accept | VER-5.6 | NUM.MISSING_ADJUSTABLE_DT | No check failed. |
| Compressible cavity cfl blowup | faulted | Reject | VER-5.2 | NUM.MISSING_ADJUSTABLE_DT | run.crash · value SIGFPE |
| Compressible cavity missing dict | faulted | Reject | VER-5.2 | ENV.MISSING_DICT | run.crash · value FOAM FATAL ERROR |
| Backward-facing step healthy | healthy | Accept | VER-5.6 | — | bounds.turbulence · value 4.0 |
| Backward-facing step mass imbalance | faulted | Reject | VER-5.4 | NUM.SOLVER_TOLERANCE | residual.not_converged · value 0.0097967 · threshold 0.001 |
| Backward-facing step oscillation | faulted | Reject | VER-5.2 | NUM.OVER_RELAXATION | run.crash · value SIGFPE |
| Backward-facing step over relaxation | faulted | Reject | VER-5.2 | NUM.OVER_RELAXATION | run.crash · value SIGFPE |
| Backward-facing step overconstrained pu | faulted | Reject | VER-5.4 | BC.OVERCONSTRAINED_PU | bc.overconstrained |
| Backward-facing step residual plateau | faulted | Reject | VER-5.4 | CONV.RESIDUAL_PLATEAU | residual.not_converged · value 0.00186427 · threshold 0.001 |
| Backward-facing step unbounded scheme | faulted | Reject | VER-5.3 | NUM.UNBOUNDED_SCHEME | bounds.turbulence · value 208.0 |
The 31 records as JSON: /data/verdicts.json
Written down before the run.
| Hypothesis | Statement | Threshold | Observed | Result |
|---|---|---|---|---|
| RM-01.H1 | at most 2 false accepts over the faulted runs | ≤ 2 | 0 of 23 | supported |
| RM-01.H2 | zero false alarms over the healthy runs | == 0 | 0 of 8 | supported |
| RM-02.H1.held_in | fault in the top two on at least 18 held-in faulted tasks | ≥ 18 | 21 of 21 | supported |
| RM-02.H1.held_out | fault in the top two on at least 3 held-out faulted tasks | ≥ 3 | 2 of 2 | refuted |
The held-out hypothesis is recorded as refuted, not re-scored. Its threshold was set when two solver families were held out; with one, only two faulted runs remain, so three could not be reached. Both of the two were ranked correctly.
Point it at a folder of past runs.
Become a design partner→
Open source · free