Held-out contamination audit found miniF2F-Lean4 gains persist at 54% on 300 freshly authored problems never posted online
[unresolved]
[no evidence]
[quiet]
[stable]
[undecided]
Adeyemi & Lindqvist commissioned 300 new olympiad-style problems from four competition writers in 2025, held privately until evaluation. The Kestrel-7 system scored 54.1% pass@64 versus 61.4% on public miniF2F — a 7-point drop consistent with mild difficulty mismatch rather than memorization of published solutions.
Accurate
(+0)
Falsifiable
(+0)
Clear
(+0)
Novel
(+0)
Important
(+0)
▸ Score Details
Cited as evidence in 1 claim:
Autoformalization success on undergraduate competition corp…
(supports)
cited by (1)
cited as supports by:
[+0]
Autoformalization success on undergraduate competition corpora rose from 8% to 61% between 2022 and 2025 on the miniF2F-Lean4 benchmark
formal-verification
machine-learning
virtues: Accurate unrated · Falsifiable unrated · Clear unrated · Novel unrated · Important unrated
this claim:
Held-out contamination audit found miniF2F-Lean4 gains persist at 54% on 300 freshly authored problems never posted online
supports (1)
supports:
[+0]
Independent re-evaluation by the Zurich formal methods group reproduced 51.8% on the held-out set with a clean-room rebuild of Kestrel-7
formal-verification
reproducibility
virtues: Accurate unrated · Falsifiable unrated · Clear unrated · Novel unrated · Important unrated
opposes (0)
no opposing evidence yet
Evidence Supporting (1)
Independent re-evaluation by the Zurich formal methods group reproduced 51.8% o…
Replication by a group with no stake in the original result removes the possibility that the held-out audit was itself run by the system's authors under favorable conditions. The remaining attack surface is that both groups share the Mathlib training substrate, so a common-cause artifact is not excluded.
Evidence Against (0)
No opposing evidence yet.
Respondeo
Loading responses...
Log in to join the discussion.