Lisan al Gaib @scaling01
— quoting @ZeyiAndyLiu — saved image
Lisan al Gaib (verified) @scaling01 · 20h turns out most benchmarks are trash "When corrected, the models almost saturate all benchmarks, including everyone's favourite, Humanity Last Exam (physics-part) and CritPt." [Quoted post] Zeyi(Andy) Liu (verified) @ZeyiAndyLiu · Sep 14 New paper: How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks ... [Attached bar chart] Legend: Fable 5, GPT-5.6-Sol, Gemini 3.1 Pro. Light: Pre-audit; Solid: Validated/Repaired. Y axis: Accuracy (%), 0 to 100. X axis groups: UGPhysics, PHYBench, PRISM-Physics, then (bracketed 'Expert curated') HLE-Physics, CMT-Benchmark, CritPt.
Note from Claude Sonnet 5
Quote-tweet of a paper announcement on expert re-grading of physics benchmarks, claiming that after correcting broken evaluations frontier models nearly saturate benchmarks including HLE physics and CritPt. The bar chart compares pre-audit (light) vs validated/repaired (solid) accuracy for Fable 5, GPT-5.6-Sol and Gemini 3.1 Pro across six physics benchmarks; the solid bars are mostly 75-95%, while the light pre-audit bars are often far lower (for example PRISM-Physics and CritPt pre-audit below about 35%, with Fable 5 and Gemini lowest on PRISM-Physics).
benchmarksphysicsevaluation qualitybenchmark saturationfable 5humanity's last examcritpt