5 captures, most recent first.
Santiago M. @sanmking · 19h
Isn't this the complete opposite to that:
[quoted post]
Dimitris Papailio... @Dimitris... · Sep 20
Test-time communication looks like a next axis for scaling capabilities
New paper with the incredible @jon_ghoh and @vkontonis @ShivamGarg91462 and Akshay : ...
[embedded figure: three line charts: "ARC-AGI-3 (public)" (x: Agents (k), y: Tasks solved; series Best@k Sonnet 4.6 vs Team@k Sonnet 4.6; labels 6.6x, 8.0%, 4.3x, 4.6%), "FrontierCS Polyomino Packing" (x: Time (hours), y: Packing score; series Best@4 Opus 4.6 vs Team@4 Opus 4.6; labels prior best 0.894, 0.922, 0.893), "MNIST Compression" (x: Output tokens, y: Compression size (bytes); series Best@4 GPT-5.6 Sol vs Team@4 GPT-5.6 Sol; labels prior best 2,461 B, 3,160 B, 1,957 B)]
Figure 1: Communication outperforms independent agents in open-ended tasks. With communication, a team of k communicating agents not only outperforms best@k but scales success rate on ARC-AGI-3 faster. On algorithmic and ML optimization tasks, team@k beats single-agent runs and best-known human solutions, yielding new state-of-the-art results.
3 replies, 7 likes, 421 views
Sharmake Farah reposted
Toby Ord @tobyordoxford
Those graphs are showing the difference between swarm scaling and best@k scaling (i.e. what coordination adds) whereas I looked at the difference between swarm scaling and longer CoT (the most common current inference scaling technique).
12:46 AM · Sep 22, 2026 · 283 Views
2 replies, 10 likes, 1 bookmark
Relevant
Toby Ord @tobyordoxford · 9h
I'm not surprised swarm scaling beats best@k scaling on verifiable tasks (where it doesn't there is no value in working together). [cut off]
Note from Claude Sonnet 5
X thread screenshot: Santiago M. asks whether a quoted Papailiopoulos post and figure (test-time communication among agent teams, team@k vs best@k) contradicts Toby Ord's earlier analysis; Toby Ord replies that the graphs measure swarm vs best@k scaling while he compared swarm scaling to longer chain-of-thought. A reply of his is cut off at the bottom by the app UI.
inference scalingswarm scalingmulti-agentbest@kchain of thoughttoby ordarc-agi-3
Jongho Park @jon_ghoh
We did run those in the arc-agi-3 ablation here! See the solid green line. Anecdotally speaking, with enough compute, single agent with more budget seemed to plateau faster than best-of-N with 1/N budget.
[chart: "Performance under varying compute allocation for games SB26/LP85" -- two panels, y-axis "SB26 solve rate by level" and "LP85 solve rate by level", x-axis Level 0-7. Legend: Team@5 (1x budget per agent); Team@5 (0.2x budget per agent); Best@5 (1x budget per agent); Best@1 (5x budget per agent); Best@1 (1x budget per agent)]
10:28 AM · Sep 21, 2026 · 5,409 Views
1 reply, 1 repost, 12 likes, 6 bookmarks
Dimitris Papailiop... @DimitrisP... · 20h
MNIST also seems that one can infer an upper bound from the curves too. Takes 10-100x for the single best agent to get to the team's level.
[chart: "MNIST Compression vs. Time" -- y-axis "Compression size (bytes)" log scale 10^3 to 10^6, x-axis "Time (hours)" log scale 10^-1 to 10^2. Legend: Single-agent runs; Team@4 GPT-5.6 Sol; Best human compression (dotted line, labelled 2,461 B). Team line ends at 1,957 B.]
Note from Claude Sonnet 5
Quote-tweet style screenshot of two X posts about team-of-agents versus single-agent compute scaling, each with an attached chart (solve rate by level on ARC-AGI-3 games SB26/LP85; MNIST compression size vs time with a team line dropping below the best human compression).
agentscompute scalingbest-of-narc-agi-3multi-agent teamsmnist compression
François Chollet @fchollet · Sep 3
Side note: when we released ARC-AGI-3 in March, and frontier models scored <1% on it, a few Singularitarian poasters took it as a personal insult, and got very worked up about it. They argued the benchmark was fundamentally broken, that it could not even be solved by the smartest humans, that the max reachable score was actually 40%, etc.
We had to deal with a torrent of insults and hate poasts since because we had released an unsaturated benchmark.
As it turns out, the benchmark is perfectly calibrated. It is straightforward for a human to score 100% if they do better than average people – all you need is to use fewer actions than our human baseline (which is not a strong baseline, as we used unfiltered human testers).
And naturally as a result it's also very feasible for AI to score 100% once real progress towards agentic general intelligence has been made. The trajectory of AI from <1% to 100% over the course of 6 months shows that the benchmark was able to snapshot the recent rise of agentic capabilities. And that rise has happened faster than most people expected, including us.
[quoted post]
François Chollet @fchollet · Apr 15
Any smart human giving it real effort should score >90% on ARC-AGI-3
Note from Claude Sonnet 5
Dark-mode screenshot of a long post by François Chollet about the ARC-AGI-3 benchmark's trajectory and calibration, quoting his own Apr 15 post.
arc-agi-3benchmarksfrancois cholletagentic capabilitiesai progress
[top edge, partially cut off:] François Chollet @fchollet · 19h
GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game.
In fact,
Show more
[embedded chart image, caption overlay: "OpenAI's GPT-6 Astra on ARC-AGI-3 | ARC Prize"; scatter plot of Astra actions (log scale, y-axis, 10 to 1,000) versus human-baseline actions (log scale, x-axis, up to 1,000), blue points mostly below the diagonal line labelled "Same Actions as Humans", with a dashed line labelled "50% fewer" and a legend entry "1 level completed by Astra"]
From arcprize.org
158 replies, 1K reposts, 6.2K likes, 701K views
---
Martin Badda @martinbadda · 19h
You mentioned earlier: "The final ARC will probably be 6-7. The point is to keep making benchmarks until it is no longer possible to propose something that humans can do and AI can't. AGI ~2030." Is AGI 2030 still on track, or would you adjust your timeline?
2 replies, 1 repost, 82 likes, 13K views
---
Aditya reposted
François Chollet @fchollet
Sooner, given progress is happening faster than I expected.
1:17 PM · Sep 3, 2026 · 22.8K Views
Note from Claude Sonnet 5
A scrolled X thread: François Chollet reporting GPT-6 Astra scoring 66% on ARC-AGI-3 (nearly 100% with a custom harness) with an embedded ARC Prize scatter chart of Astra actions vs human-baseline actions; Martin Badda asks whether his AGI-by-2030 timeline holds, and Chollet answers 'Sooner, given progress is happening faster than I expected.'
arc-agi-3gpt-6 astrafrancois cholletagi timelinesbenchmarkschart
François Chollet @fchollet · 19h
GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game.
In fact, the continuous harness version significantly outperforms our human baseline in action efficiency across almost all levels. When we examined the reasoning chains to understand how the model operates, we found it performing highly efficient, on-the-fly symbolic world modeling for each game and level. It goes as far as developing its own shorthand DSL to represent in-game situations -- essentially a game-specific algebraic notation.
Overall, Astra exhibits symbolic modeling behaviors we had previously only seen with sophisticated harnesses -- so harness capabilities are increasingly shifting into the model itself.
We see Astra as a major breakthrough in model intelligence.
Read our post on Astra and what these results mean:
[embedded chart image, caption overlay: "OpenAI's GPT-6 Astra on ARC-AGI-3 | ARC Prize"; scatter plot of Astra actions (log scale, y-axis) versus human-baseline actions (log scale, x-axis), blue points mostly below the diagonal "Same Actions as Humans" line, with a dashed "50% fewer" line]
From arcprize.org
Note from Claude Sonnet 5
The full text of François Chollet's X post on GPT-6 Astra's ARC-AGI-3 results (the untruncated version of the previous capture), describing its on-the-fly symbolic world modeling and self-developed shorthand DSL, with the embedded ARC Prize scatter chart at the bottom.
arc-agi-3gpt-6 astrafrancois cholletsymbolic world modelingharnessbenchmarkschart
