← All topics

arc agi

2 captures, most recent first.

Noam Brown @polynoamial

— replying to an Inverse Gary parody account — saved image

Inverse Gary ... [verified] @Inve... · Sep 14, 2024
[Parody account]
how do you feel about arc agi

1 reply   18 likes   7.2K views

norvid_studies reposted

Noam Brown [verified] @polynoamial

It's a good benchmark. I would bet general models will exceed human performance on it within two years.

7:00 PM · Sep 14, 2024 · 75.8K Views
Note from Claude Sonnet 5

Continuation of the previous Noam Brown thread: a parody account asks how he feels about ARC-AGI, and Noam Brown answers (Sep 14, 2024) that it is a good benchmark and he would bet general models will exceed human performance on it within two years.

arc agibenchmarksnoam brownpredictionhuman-level performance

Noam Brown @polynoamial

— quoting @WenhuChen; reply by an Inverse Gary parody account — saved image

Noam Bro... [verified] @polynoa... · Sep 14, 2024

The AI field desperately needs harder evals that take into consideration continued fast progress.

[quoted post]
Wenhu Chen @WenhuChen · Sep 13, 2024

Crazy bump of o1-review on MMLU-Pro math subtask! It brings the previous highest score from 79% to 91%. I am still waiting the other tasks as my api quota for o1 is pretty low.
This result also confirms the annotation quality...

[embedded table image]
MMLU-Pro Leaderboard

Models | Data Source | Math
o1-preview | TIGER-Lab | 0.9104
GPT-4o (2024-08-06) | TIGER-Lab | 0.7942
Grok-2 | Self-Reported | 0.7927
Claude-3.5-Sonnet | TIGER-Lab | 0.7683
GPT-4o (2024-05-13) | TIGER-Lab | 0.7609
Grok-2-mini | Self-Reported | 0.7609
DeepSeek-Chat-V2_5 | TIGER-Lab | 0.7535
Gemini-1.5-Pro | Self-Reported | 0.7276
GPT-4o-mini | TIGER-Lab | 0.7232
Qwen2-72B-Chat | TIGER-Lab | 0.7098
Claude-3-Opus | TIGER-Lab | 0.6957
magnum-72b-v1 | TIGER-Lab | 0.6737
Yi-large | TIGER-Lab | 0.6481

28 replies   52 reposts   506 likes   89K views

[reply]
Inverse Gary ... [verified] @Inve... · Sep 14, 2024
[Parody account]
how do you feel about arc agi

1 reply   18 likes   7.2K views

[next item, cut off at bottom of screen]
norvid_studies reposted
Noam Brown [verified] @polynoamial [cut off]
Note from Claude Sonnet 5

Thread screenshot from Noam Brown's X profile showing a Sep 14, 2024 post that the field needs harder evals accounting for fast progress, quoting Wenhu Chen's MMLU-Pro leaderboard table where o1-preview scores 0.9104 on math, with a parody-account reply asking about ARC-AGI. Bottom partly covered by the nav bar and a floating button.

evalsmmlu-proo1-previewbenchmark saturationnoam brownarc agileaderboard