Held-out contamination audit found miniF2F-Lean4 gains persist at 54% on 300 freshly authored problems never posted online

open — on 1 supporting to 0 opposing, weighted 0.0 · minimal · what this means

no opposing evidence — this claim has not been tested · no sources · falsifiability unrated

0 ◎ benchmark_hygiene (3) · 2 months ago · machine-learning, formal-verification

Adeyemi & Lindqvist commissioned 300 new olympiad-style problems from four competition writers in 2025, held privately until evaluation. The Kestrel-7 system scored 54.1% pass@64 versus 61.4% on public miniF2F — a 7-point drop consistent with mild difficulty mismatch rather than memorization of published solutions.

rate this claim
  • Accurate 0
  • Falsifiable 0
  • Clear 0
  • Novel 0
  • Important 0

evidence

cited by (1)
[+0] Autoformalization success on undergraduate competition corpora rose from 8% to 61% between 2022 and 2025 on the miniF2F-Lean4 benchmark discuss
this claim
supports (1)
[+0] Independent re-evaluation by the Zurich formal methods group reproduced 51.8% on the held-out set with a clean-room rebuild of Kestrel-7 discuss reproducibility
whole evidence graph shown depth 1 2 3 both pro con full
what the marks mean
  • supports · opposes · cites this claim · the claim you are reading
  • [+9] net votes on the evidence link — whether this is good evidence here, not whether the claim itself is true
  • Af··· accurate, falsifiable, clear, novel, important — a capital is established, lowercase weak, unrated; no strip at all where nothing has been rated yet
  • [?] no score has been computed for that claim yet
  • field shown only where a claim's fields differ from the one above it, so a field marks where the argument crosses into another

respondeo

Loading responses… open them

discussion

Log in to join the discussion.