Håvard Ihle @htihle · 8h
GPT 6 Astra Pro (max) and GPT 6 Astra (max) scores 93.6% and 93.3% on WeirdML, the two highest scores.
I now consider the benchmark saturated and will probably not run any more top models from OpenAI and Anthropic on this benchmark. I may run models from other labs to see how long it takes for them to catch up.
Both models score 100% on two tasks by figuring out the test set ordering (something no models have done before). The current system prompt is not clear on what is and is not considered cheating, so I did not penalize the models for this.
[embedded image: "WeirdML Results Summary" table, columns MODEL, AVERAGE ACCURACY ACROSS 17 TASKS, COST / RUN (USD), CODE LENGTH (#LINES), OUTPUT TOKENS; rows partly legible:]
1 gpt-6-astra-pro (max): 0.936, cost [illegible], 521 lines, [illegible] tokens
2 gpt-6-astra (max): 0.933, cost [illegible], 485 lines, 21 941 tokens
3 claude-fable-5.1 (max): 0.929, cost [illegible], 251 lines, 31 644 tokens
4 gpt-6-astra (high): 0.929, cost [illegible], 299 lines, 8 056 tokens
5 claude-fable-5.1 (high): 0.923, cost [illegible], 150 lines, 8 368 tokens
6 claude-fable-5 (max): 0.919, cost [illegible], 218 lines, 22 153 tokens
7 claude-opus-5 (max): 0.918, cost [illegible], 245 lines, 25 400 tokens
8 claude-opus-5 (high): [cut off]
Note from Claude Sonnet 5
Tweet by Håvard Ihle reporting GPT 6 Astra Pro (max) and GPT 6 Astra (max) at 93.6% and 93.3% on the WeirdML benchmark, which he now considers saturated, and noting both models reached 100% on two tasks by inferring the test-set ordering. Below the text is the top of the WeirdML Results Summary leaderboard table (GPT 6 Astra variants ranked above Claude Fable 5.1, Fable 5 and Opus 5 variants, with per-run cost, code length and output-token bars). Public commentary on benchmark saturation and cheating behaviour; no technical exploit detail.
Håvard Ihle @htihle · 14h
GPT 6 Astra (high) scores 92.9% on WeirdML and matches Fable 5.1 (max) for the top score.
It sets a new individual high score on 6 of the 17 tasks, and is overall very solid.
Interestingly it writes significantly less code than Sol and other recent GPT models, although still more than Claude.
I will do a more careful analysis when the max and pro max runs are done.
Note from Claude Sonnet 5
Dark-mode X post by Håvard Ihle reporting GPT 6 Astra (high) scoring 92.9% on WeirdML, tying Fable 5.1 (max) for top score, with notes on per-task highs and code length.
Håvard Ihle @htihle · 13h
Claude opus 4.6 (adaptive) takes the lead on WeirdML with 77.9% ahead of gpt-5.2 (xhigh) at 72.2%.
It sets a new high score on 3 tasks including scoring 73% on the hardest task (digits_generalize) up from 59%.
Opus 4.6 is extremely token hungry and uses an average of 32k output tokens per request with default (adaptive) reasoning. Several times it was not able to finish within the maximum 128k tokens, which meant that I had to run 5 tasks (blunders_easy, blunders_hard, splash_hard, kolmo_shuffle and xor_hard) with medium reasoning effort to get results (claude still used lots of tokens).
Because of the high cost, opus 4.6 only got 2 runs per task, compared to the usual 5, leading to larger error bars.
[Chart 1: "WeirdML Results Summary" — bar chart of average accuracy across 17 tasks and cost per run (USD) for various models, top score 0.903, Claude Opus entries around 0.779/0.722/0.699 etc. with costs ranging ~$0.22–$7.65]
[Chart 2: "WeirdML: Interactive Model Comparison" scatter plot, average cost per run (USD, log scale) vs performance, showing models from OpenAI, Anthropic, Google, DeepSeek, xAI, Meta, Mistral, Qwen, Moonshot]
> QUOTED: Håvard Ihle @htihle · Jun 27, 2025
> WeirdML v2 is now out! The update includes a bunch of new tasks (now 19 tasks total, up from 6), and results from all the latest models. We now also track api costs and other ...
Note from Claude Sonnet 5
Benchmark results from the WeirdML evaluation suite showing Claude Opus 4.6 taking the top score (77.9%) ahead of GPT-5.2, with a note on Opus 4.6's high token/cost usage. Relevant to tracking frontier model capability comparisons over time, adjacent to the project's interest in Claude-generation capability jumps.
