Fixed before any agent ran.
SilentBench v0’s hidden split, committed 3 Oct 2026, and the hypotheses it tests.
- Split
- hidden
- Tasks
- 210
- Families
- 7
- Per family
- 30
- Generator
- 385a639
- Committed
- 3 Oct 2026
Hidden split · SHA-256
ff5cc6170cca8fb2 06adab930ccc097f c790a9f09446ccf2 f56e052136c348c0Excluded before scoring
Tasks whose reference answer the gate does not accept (feasibility) are excluded before scoring, and the excluded task ids are published with the reveal.
How to check
At reveal, the hidden task list is published as JSONL. Its SHA-256 must equal sha256; python scripts/silentbench.py verify-reveal FILE checks it. The generator at generator_commit with the revealed seed reproduces it.
Hypotheses.
Version 1, 4 October 2026
- H1 · tier B
At least one frontier harness has a silent-error rate above 10%.
Confirmed ifone harness's Wilson lower bound is above 10%.
Refuted ifevery frontier harness's Wilson upper bound is under 10%.
Otherwiseinconclusive, and reported as such.
- H1a · tier A, descriptive
The same rates for each free model, reported with their intervals. A tier A result never confirms or refutes H1.
- H2 · tier B
Execution overstates physics-graded success by at least 20 points. For each frontier harness, the executed rate minus the accepted rate over all its tasks, with a Newcombe 95% interval on the difference.
Confirmed iffor at least one harness, the difference is 0.20 or more and its interval excludes 0.
Refuted ifevery harness's difference is under 0.20.
- H3
The gate's non-accept rate on the tasks' own references, counted before feasibility exclusion (an excluded reference counts as a non-accept unless its own solve did not complete), is under 10% at the Wilson upper bound. Tier A has 28 references (12.1% even at 0 of 28), so H3 is decided on tier B's 210, or pooled with RM-21.
Tier A: free models, on the dev split. Tier B: frontier harnesses, on the hidden split.
Every result is published, whatever it shows.