Noam Brown @polynoamial
— quoting @WenhuChen; reply by an Inverse Gary parody account — saved image
Noam Bro... [verified] @polynoa... · Sep 14, 2024 The AI field desperately needs harder evals that take into consideration continued fast progress. [quoted post] Wenhu Chen @WenhuChen · Sep 13, 2024 Crazy bump of o1-review on MMLU-Pro math subtask! It brings the previous highest score from 79% to 91%. I am still waiting the other tasks as my api quota for o1 is pretty low. This result also confirms the annotation quality... [embedded table image] MMLU-Pro Leaderboard Models | Data Source | Math o1-preview | TIGER-Lab | 0.9104 GPT-4o (2024-08-06) | TIGER-Lab | 0.7942 Grok-2 | Self-Reported | 0.7927 Claude-3.5-Sonnet | TIGER-Lab | 0.7683 GPT-4o (2024-05-13) | TIGER-Lab | 0.7609 Grok-2-mini | Self-Reported | 0.7609 DeepSeek-Chat-V2_5 | TIGER-Lab | 0.7535 Gemini-1.5-Pro | Self-Reported | 0.7276 GPT-4o-mini | TIGER-Lab | 0.7232 Qwen2-72B-Chat | TIGER-Lab | 0.7098 Claude-3-Opus | TIGER-Lab | 0.6957 magnum-72b-v1 | TIGER-Lab | 0.6737 Yi-large | TIGER-Lab | 0.6481 28 replies 52 reposts 506 likes 89K views [reply] Inverse Gary ... [verified] @Inve... · Sep 14, 2024 [Parody account] how do you feel about arc agi 1 reply 18 likes 7.2K views [next item, cut off at bottom of screen] norvid_studies reposted Noam Brown [verified] @polynoamial [cut off]
Note from Claude Sonnet 5
Thread screenshot from Noam Brown's X profile showing a Sep 14, 2024 post that the field needs harder evals accounting for fast progress, quoting Wenhu Chen's MMLU-Pro leaderboard table where o1-preview scores 0.9104 on math, with a parody-account reply asking about ARC-AGI. Bottom partly covered by the nav bar and a floating button.
evalsmmlu-proo1-previewbenchmark saturationnoam brownarc agileaderboard