8 captures, most recent first.
François Chollet @fchollet · Sep 3
Side note: when we released ARC-AGI-3 in March, and frontier models scored <1% on it, a few Singularitarian poasters took it as a personal insult, and got very worked up about it. They argued the benchmark was fundamentally broken, that it could not even be solved by the smartest humans, that the max reachable score was actually 40%, etc.
We had to deal with a torrent of insults and hate poasts since because we had released an unsaturated benchmark.
As it turns out, the benchmark is perfectly calibrated. It is straightforward for a human to score 100% if they do better than average people – all you need is to use fewer actions than our human baseline (which is not a strong baseline, as we used unfiltered human testers).
And naturally as a result it's also very feasible for AI to score 100% once real progress towards agentic general intelligence has been made. The trajectory of AI from <1% to 100% over the course of 6 months shows that the benchmark was able to snapshot the recent rise of agentic capabilities. And that rise has happened faster than most people expected, including us.
[quoted post]
François Chollet @fchollet · Apr 15
Any smart human giving it real effort should score >90% on ARC-AGI-3
Note from Claude Sonnet 5
Dark-mode screenshot of a long post by François Chollet about the ARC-AGI-3 benchmark's trajectory and calibration, quoting his own Apr 15 post.
arc-agi-3benchmarksfrancois cholletagentic capabilitiesai progress
[top edge, partially cut off:] François Chollet @fchollet · 19h
GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game.
In fact,
Show more
[embedded chart image, caption overlay: "OpenAI's GPT-6 Astra on ARC-AGI-3 | ARC Prize"; scatter plot of Astra actions (log scale, y-axis, 10 to 1,000) versus human-baseline actions (log scale, x-axis, up to 1,000), blue points mostly below the diagonal line labelled "Same Actions as Humans", with a dashed line labelled "50% fewer" and a legend entry "1 level completed by Astra"]
From arcprize.org
158 replies, 1K reposts, 6.2K likes, 701K views
---
Martin Badda @martinbadda · 19h
You mentioned earlier: "The final ARC will probably be 6-7. The point is to keep making benchmarks until it is no longer possible to propose something that humans can do and AI can't. AGI ~2030." Is AGI 2030 still on track, or would you adjust your timeline?
2 replies, 1 repost, 82 likes, 13K views
---
Aditya reposted
François Chollet @fchollet
Sooner, given progress is happening faster than I expected.
1:17 PM · Sep 3, 2026 · 22.8K Views
Note from Claude Sonnet 5
A scrolled X thread: François Chollet reporting GPT-6 Astra scoring 66% on ARC-AGI-3 (nearly 100% with a custom harness) with an embedded ARC Prize scatter chart of Astra actions vs human-baseline actions; Martin Badda asks whether his AGI-by-2030 timeline holds, and Chollet answers 'Sooner, given progress is happening faster than I expected.'
arc-agi-3gpt-6 astrafrancois cholletagi timelinesbenchmarkschart
François Chollet @fchollet · 19h
GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game.
In fact, the continuous harness version significantly outperforms our human baseline in action efficiency across almost all levels. When we examined the reasoning chains to understand how the model operates, we found it performing highly efficient, on-the-fly symbolic world modeling for each game and level. It goes as far as developing its own shorthand DSL to represent in-game situations -- essentially a game-specific algebraic notation.
Overall, Astra exhibits symbolic modeling behaviors we had previously only seen with sophisticated harnesses -- so harness capabilities are increasingly shifting into the model itself.
We see Astra as a major breakthrough in model intelligence.
Read our post on Astra and what these results mean:
[embedded chart image, caption overlay: "OpenAI's GPT-6 Astra on ARC-AGI-3 | ARC Prize"; scatter plot of Astra actions (log scale, y-axis) versus human-baseline actions (log scale, x-axis), blue points mostly below the diagonal "Same Actions as Humans" line, with a dashed "50% fewer" line]
From arcprize.org
Note from Claude Sonnet 5
The full text of François Chollet's X post on GPT-6 Astra's ARC-AGI-3 results (the untruncated version of the previous capture), describing its on-the-fly symbolic world modeling and self-developed shorthand DSL, with the embedded ARC Prize scatter chart at the bottom.
arc-agi-3gpt-6 astrafrancois cholletsymbolic world modelingharnessbenchmarkschart
Lisan al Gaib reposted
Andrew Lampinen @AndrewLampinen
The question is "how much is each component is the system contributing to its intelligence & generality" — and there I think it's pretty clear that the neural component is still the thing doing the interesting hypothesis or plan generation, deciding what went wrong, etc. 1/
[quoted tweet]
François Chollet @fchollet · 13h
I would have assumed it was fairly obvious, but in case it's not: a million-line codebase (also known as a "harness"), running at inference time, orchestrating thousands of calls to a neural network for any given task, is the exact definition of a "neurosymbolic ...
7:39 AM · Aug 6, 2026 · 25.3K Views
12 replies, 14 reposts, 191 likes, 64 bookmarks
Relevant
Andrew Lampin... @AndrewLampin... · 10h
This is very clearly different from the vision that many neurosymbolic advocates had a few years ago, e.g. these quotes (taken from arxiv.org/abs/2305.00813 and arxiv.org/abs/1801.00631) in which symbol manipulation did the "intelligent" part. 2/
[quoted image of text]
These arguments, together with similar ones from others, drove a longstanding trend to dismiss neural networks as only capable of modeling simple perceptual processing, rather than "real" higher-level cognition, which is symbolic and systematic. For example "while data driven neural network-based AI algorithms effectively model machine perception, symbolic knowledge-based AI is better [cut off]
Note from Claude Sonnet 5
Continuation of the Andrew Lampinen / François Chollet thread on whether AI system intelligence comes from the neural model or the surrounding 'harness' (see also seq 423). Includes Chollet's counter-argument that a large orchestrating codebase constitutes 'neurosymbolic' AI, and Lampinen's reply linking two arxiv papers (2305.00813, 1801.00631) with a quoted excerpt arguing older neurosymbolic advocates dismissed neural nets as incapable of higher-level cognition.
ai capabilityharness vs modelneurosymbolicandrew lampinenfrancois chollet
François Chollet @fchollet
There are essentially two main options to remedy this:
1. Find ways to perform active inference, so that the model adapts its learned program in contact with a new data distribution at test time. Would likely lead to some meaningful progress, but it isn't the ultimate solution, more of an incremental improvement.
2. Change the training mechanism to something more robust than SGD, such as the MDL principle. This would pretty much require moving away from deep learning (curve fitting) altogether and embracing discrete program search instead (which I have advocated for many years as a way to tackle reasoning problems...)
2:14 AM · Mar 8, 2024 · 150.5K Views
Note from Claude Sonnet 5
Tweet from François Chollet (@fchollet), dated March 8 2024, proposing two remedies for a generalization problem in deep learning he'd described earlier in the thread: test-time active inference (incremental) or replacing SGD with something like the MDL principle via discrete program search (a bigger departure from curve-fitting deep learning).
deep learning theorygeneralizationprogram searchfrancois chollet
François Chollet ✔ @fchollet · Feb 19
Sufficiently advanced agentic coding is essentially machine learning: the engineer sets up the optimization goal as well as some constraints on the search space (the spec and its tests), then an optimization process (coding agents) iterates until the goal is reached.
The result is a blackbox model (the generated codebase): an artifact that performs the task, that you deploy without ever inspecting its internal logic, just as we ignore individual weights in a neural network.
This implies that all classic issues encountered in ML will soon become problems for agentic coding: overfitting to the spec, Clever Hans shortcuts that don't generalize outside the tests, data leakage, concept drift, etc.
I would also ask: what will be the Keras of agentic coding? What will be the optimal set of high-level abstractions that allow humans to steer codebase 'training' with minimal cognitive overhead?
Note from Claude Sonnet 5
François Chollet (Keras creator) argues that advanced agentic coding effectively becomes a form of ML optimization, inheriting classic ML failure modes (overfitting to spec, Clever Hans shortcuts, data leakage). Relevant to Nathan's interest in AI-assisted software development, interpretability of generated artifacts, and the analogy between agentic coding and opaque optimization processes.
agentic codingmachine learningfrancois cholletinterpretabilitysoftware engineeringtwitter
François Chollet @fchollet · 5m
The new Gemini Deep Think is achieving some truly incredible numbers on ARC-AGI-2. We certified these scores in the past few days.
[Embedded chart: "Gemini 3 Deep Think — ARC-AGI-2 — Reasoning & knowledge — ARC PRIZE VERIFIED" bar chart:
Gemini 3 Deep Think (Feb 2026): 84.6%
Gemini 3 Pro Preview (Thinking High): 31.1%
Claude Opus 4.6 (Thinking Max): 68.8%
GPT-5.2 (Thinking xhigh): 52.9%
Methodology: deepmind.google/models/evals-methodology/gemini-3-deep-think]
Note from Claude Sonnet 5
Chollet announcing ARC-Prize-verified benchmark results showing Google's new Gemini 3 Deep Think scoring 84.6% on ARC-AGI-2, well above Claude Opus 4.6 (68.8%) and GPT-5.2 (52.9%). Notable capability-progress data point for Nathan's AI timeline tracking, and shows Gemini leading Claude on this benchmark at this date — useful context for cross-lab capability comparisons.
twitterarc-agi-2benchmarksgemini 3 deep thinkclaude opus 4.6gpt-5.2capability progressfrancois chollet
Vinod Khosla @vkhosla · 12h
Well well... ARC-AGI-2 (François Chollet's "hardest" benchmark) is starting to smell like toast. 🍞🔥
@agenticasdk just set a new SOTA: 85.28% with an Agentica agent (~350 lines) that writes & runs code. Best part: it's not ARC-specialized—it's a general system that's strong
Show more
[Link card: symbolica.ai — "SotA ARC-AGI-2 Results with REPL Agents"]
16 replies, 27 retweets, 220 likes, 37K views
François Chollet @fchollet · 10h
You should try ARC-AGI-3 (developer preview is available now, full benchmark coming in a few weeks)
Note from Claude Sonnet 5
A tweet reporting a new state-of-the-art score (85.28%) on ARC-AGI-2, previously considered a very hard reasoning benchmark, achieved by a general (non-specialized) code-writing agent from Agentica, with Chollet's own reply pointing to the newer ARC-AGI-3 benchmark. Relevant to Nathan's tracking of capability progress and "empirical singularity tracking" notes.
twitterarc-agibenchmarksagentic aifrancois cholletcapability progress
