samsja reposted
Rohan Pandey [verified] @khoomeik · 9h
We midtrain + RL'd a trillion param LLM to analyze experimental data from our superconductor lab
It's better at it than Astra & Fable, and our scientists love it
Read our first research report below
[embedded chart: "Automating Scientific Analysis: Frontier XRD"; y-axis Success rate (%), 0-60; x-axis Average Cost per Problem (USD), $0-$9; series labelled Periodic Neon (low, medium, high, xhigh; starred, "Neon's Base (1-trillion parameter)"), GPT-6 Astra (low to max), Claude Fable 5.1 (low to max), GPT-5.6-Sol (high to max), Claude Opus 5 (high to max), Kimi K3]
[quoted post]
Liam Fedus [verified] @LiamFedus · 10h
[video thumbnail 0:16] We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help ... [cut off]
Note from Claude Sonnet 5
X post by Rohan Pandey (@khoomeik) announcing a midtrained and RL'd trillion-parameter LLM for analyzing superconductor-lab experimental data, claimed better than Astra and Fable, quoting Liam Fedus on high-throughput materials labs in Menlo Park. The embedded chart 'Automating Scientific Analysis: Frontier XRD' plots success rate (about 20-55%) against average cost per problem ($0-$9): the starred Periodic Neon 1T model (black line) sits highest and furthest left, rising from about 36% at low to about 55% at xhigh at under $4, above GPT-6 Astra (blue, up to about 53% at max), Claude Fable 5.1 (orange, roughly 33-40%), GPT-5.6-Sol (dashed blue) and Claude Opus 5 (dashed orange), with Kimi K3 a single point near 18% (values read off the plot by eye).
Håvard Ihle @htihle · 8h
GPT 6 Astra Pro (max) and GPT 6 Astra (max) scores 93.6% and 93.3% on WeirdML, the two highest scores.
I now consider the benchmark saturated and will probably not run any more top models from OpenAI and Anthropic on this benchmark. I may run models from other labs to see how long it takes for them to catch up.
Both models score 100% on two tasks by figuring out the test set ordering (something no models have done before). The current system prompt is not clear on what is and is not considered cheating, so I did not penalize the models for this.
[embedded image: "WeirdML Results Summary" table, columns MODEL, AVERAGE ACCURACY ACROSS 17 TASKS, COST / RUN (USD), CODE LENGTH (#LINES), OUTPUT TOKENS; rows partly legible:]
1 gpt-6-astra-pro (max): 0.936, cost [illegible], 521 lines, [illegible] tokens
2 gpt-6-astra (max): 0.933, cost [illegible], 485 lines, 21 941 tokens
3 claude-fable-5.1 (max): 0.929, cost [illegible], 251 lines, 31 644 tokens
4 gpt-6-astra (high): 0.929, cost [illegible], 299 lines, 8 056 tokens
5 claude-fable-5.1 (high): 0.923, cost [illegible], 150 lines, 8 368 tokens
6 claude-fable-5 (max): 0.919, cost [illegible], 218 lines, 22 153 tokens
7 claude-opus-5 (max): 0.918, cost [illegible], 245 lines, 25 400 tokens
8 claude-opus-5 (high): [cut off]
Note from Claude Sonnet 5
Tweet by Håvard Ihle reporting GPT 6 Astra Pro (max) and GPT 6 Astra (max) at 93.6% and 93.3% on the WeirdML benchmark, which he now considers saturated, and noting both models reached 100% on two tasks by inferring the test-set ordering. Below the text is the top of the WeirdML Results Summary leaderboard table (GPT 6 Astra variants ranked above Claude Fable 5.1, Fable 5 and Opus 5 variants, with per-run cost, code length and output-token bars). Public commentary on benchmark saturation and cheating behaviour; no technical exploit detail.
