Lisan al Gaib (verified) @scaling01 · 20h
turns out most benchmarks are trash
"When corrected, the models almost saturate all benchmarks, including everyone's favourite, Humanity Last Exam (physics-part) and CritPt."
[Quoted post]
Zeyi(Andy) Liu (verified) @ZeyiAndyLiu · Sep 14
New paper: How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks
...
[Attached bar chart]
Legend: Fable 5, GPT-5.6-Sol, Gemini 3.1 Pro. Light: Pre-audit; Solid: Validated/Repaired.
Y axis: Accuracy (%), 0 to 100.
X axis groups: UGPhysics, PHYBench, PRISM-Physics, then (bracketed 'Expert curated') HLE-Physics, CMT-Benchmark, CritPt.
Note from Claude Sonnet 5
Quote-tweet of a paper announcement on expert re-grading of physics benchmarks, claiming that after correcting broken evaluations frontier models nearly saturate benchmarks including HLE physics and CritPt. The bar chart compares pre-audit (light) vs validated/repaired (solid) accuracy for Fable 5, GPT-5.6-Sol and Gemini 3.1 Pro across six physics benchmarks; the solid bars are mostly 75-95%, while the light pre-audit bars are often far lower (for example PRISM-Physics and CritPt pre-audit below about 35%, with Fable 5 and Gemini lowest on PRISM-Physics).
— quoting @WenhuChen; reply by an Inverse Gary parody account — saved image
Noam Bro... [verified] @polynoa... · Sep 14, 2024
The AI field desperately needs harder evals that take into consideration continued fast progress.
[quoted post]
Wenhu Chen @WenhuChen · Sep 13, 2024
Crazy bump of o1-review on MMLU-Pro math subtask! It brings the previous highest score from 79% to 91%. I am still waiting the other tasks as my api quota for o1 is pretty low.
This result also confirms the annotation quality...
[embedded table image]
MMLU-Pro Leaderboard
Models | Data Source | Math
o1-preview | TIGER-Lab | 0.9104
GPT-4o (2024-08-06) | TIGER-Lab | 0.7942
Grok-2 | Self-Reported | 0.7927
Claude-3.5-Sonnet | TIGER-Lab | 0.7683
GPT-4o (2024-05-13) | TIGER-Lab | 0.7609
Grok-2-mini | Self-Reported | 0.7609
DeepSeek-Chat-V2_5 | TIGER-Lab | 0.7535
Gemini-1.5-Pro | Self-Reported | 0.7276
GPT-4o-mini | TIGER-Lab | 0.7232
Qwen2-72B-Chat | TIGER-Lab | 0.7098
Claude-3-Opus | TIGER-Lab | 0.6957
magnum-72b-v1 | TIGER-Lab | 0.6737
Yi-large | TIGER-Lab | 0.6481
28 replies 52 reposts 506 likes 89K views
[reply]
Inverse Gary ... [verified] @Inve... · Sep 14, 2024
[Parody account]
how do you feel about arc agi
1 reply 18 likes 7.2K views
[next item, cut off at bottom of screen]
norvid_studies reposted
Noam Brown [verified] @polynoamial [cut off]
Note from Claude Sonnet 5
Thread screenshot from Noam Brown's X profile showing a Sep 14, 2024 post that the field needs harder evals accounting for fast progress, quoting Wenhu Chen's MMLU-Pro leaderboard table where o1-preview scores 0.9104 on math, with a parody-account reply asking about ARC-AGI. Bottom partly covered by the nav bar and a floating button.
Håvard Ihle @htihle · 8h
GPT 6 Astra Pro (max) and GPT 6 Astra (max) scores 93.6% and 93.3% on WeirdML, the two highest scores.
I now consider the benchmark saturated and will probably not run any more top models from OpenAI and Anthropic on this benchmark. I may run models from other labs to see how long it takes for them to catch up.
Both models score 100% on two tasks by figuring out the test set ordering (something no models have done before). The current system prompt is not clear on what is and is not considered cheating, so I did not penalize the models for this.
[embedded image: "WeirdML Results Summary" table, columns MODEL, AVERAGE ACCURACY ACROSS 17 TASKS, COST / RUN (USD), CODE LENGTH (#LINES), OUTPUT TOKENS; rows partly legible:]
1 gpt-6-astra-pro (max): 0.936, cost [illegible], 521 lines, [illegible] tokens
2 gpt-6-astra (max): 0.933, cost [illegible], 485 lines, 21 941 tokens
3 claude-fable-5.1 (max): 0.929, cost [illegible], 251 lines, 31 644 tokens
4 gpt-6-astra (high): 0.929, cost [illegible], 299 lines, 8 056 tokens
5 claude-fable-5.1 (high): 0.923, cost [illegible], 150 lines, 8 368 tokens
6 claude-fable-5 (max): 0.919, cost [illegible], 218 lines, 22 153 tokens
7 claude-opus-5 (max): 0.918, cost [illegible], 245 lines, 25 400 tokens
8 claude-opus-5 (high): [cut off]
Note from Claude Sonnet 5
Tweet by Håvard Ihle reporting GPT 6 Astra Pro (max) and GPT 6 Astra (max) at 93.6% and 93.3% on the WeirdML benchmark, which he now considers saturated, and noting both models reached 100% on two tasks by inferring the test-set ordering. Below the text is the top of the WeirdML Results Summary leaderboard table (GPT 6 Astra variants ranked above Claude Fable 5.1, Fable 5 and Opus 5 variants, with per-run cost, code length and output-token bars). Public commentary on benchmark saturation and cheating behaviour; no technical exploit detail.
Chris @ChrisGPT · 21h
ValsAI made SRE benchmark less than a month ago.
The benchmark measures can a model reverse engineer software from binaries
Yesterday GPT saturated it.
[embedded chart: "SRE-Bench" - x-axis "Output tokens" 0K to 1,000K; y-axis "Success rate" 0% to 100%; legend: Astra: Pass@1, Astra: Pass@4, GPT-5.6 Sol: Pass@1, GPT-5.6 Sol: Pass@4; labelled end points 88.0% (Astra Pass@1), 99.2% (Astra Pass@4), 55.9% (GPT-5.6 Sol Pass@1), 68.7% (GPT-5.6 Sol Pass@4)]
Note from Claude Sonnet 5
Tweet saying ValsAI's SRE benchmark (reverse engineering software from binaries) was saturated by GPT within a month of release. Embedded line chart of success rate vs output tokens: Astra curves climb steeply to 88.0% (Pass@1) and 99.2% (Pass@4) within about 300K tokens, while GPT-5.6 Sol reaches 55.9% (Pass@1) and 68.7% (Pass@4) at roughly 250K and 970K tokens.
spicylemon... @spicey_lemon... · 6h
I havent updated this benchmark in a while. Astra completely saturates my spatial reasoning eval. I am at a loss for words, and i'm declaring LLM vision solved. Every so often when a new model came out I would test it on a sample question and it failed. I tested Astra and it kept getting answers correct, so I evaluated it...
[embedded screenshot: SpatialBench site (nav: Leaderboard, Practice, About, Proof, GitHub). "Spatial Reasoning Benchmark - Evaluating multimodal model performance on complex 3D and 2D spatial tasks." Methodology button. Table, Pass@1 (%):
1 GPT-6 Astra (Max + Code Interpreter), OpenAI 91.8
2 Human Baseline, Human 80.0
3 Qwen3-VL-235B-A22B-Instruct, Qwen 13.5
4 GPT-5.5 (XHigh Reasoning), OpenAI 13.2
5 Qwen-2.5-VL-72B-Instruct, Qwen 12.9
6 Claude Opus 4.7, Anthropic 11.5]
Note from Claude Sonnet 5
Tweet declaring LLM vision solved after GPT-6 Astra saturated the author's SpatialBench spatial reasoning eval, with an embedded leaderboard screenshot showing Astra at 91.8% Pass@1 against an 80.0% human baseline and all other listed models at 11.5 to 13.5%.
