François Chollet @fchollet · 15h
Delegate coding. Never delegate understanding.
If code was previously your source of truth for managing understanding, then you need a new source of truth artifact. If code-centric workflows were how you developed that understanding, then you need new workflows.
Note from Claude Sonnet 5
Cropped X post by François Chollet about delegating coding to AI while retaining understanding, arguing for new source-of-truth artifacts and workflows.
ai codingunderstandingworkflowschollet
François Chollet @fchollet · Sep 3
Side note: when we released ARC-AGI-3 in March, and frontier models scored <1% on it, a few Singularitarian poasters took it as a personal insult, and got very worked up about it. They argued the benchmark was fundamentally broken, that it could not even be solved by the smartest humans, that the max reachable score was actually 40%, etc.
We had to deal with a torrent of insults and hate poasts since because we had released an unsaturated benchmark.
As it turns out, the benchmark is perfectly calibrated. It is straightforward for a human to score 100% if they do better than average people – all you need is to use fewer actions than our human baseline (which is not a strong baseline, as we used unfiltered human testers).
And naturally as a result it's also very feasible for AI to score 100% once real progress towards agentic general intelligence has been made. The trajectory of AI from <1% to 100% over the course of 6 months shows that the benchmark was able to snapshot the recent rise of agentic capabilities. And that rise has happened faster than most people expected, including us.
[quoted post]
François Chollet @fchollet · Apr 15
Any smart human giving it real effort should score >90% on ARC-AGI-3
Note from Claude Sonnet 5
Dark-mode screenshot of a long post by François Chollet about the ARC-AGI-3 benchmark's trajectory and calibration, quoting his own Apr 15 post.
arc-agi-3benchmarksfrancois cholletagentic capabilitiesai progress
[top edge, partially cut off:] François Chollet @fchollet · 19h
GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game.
In fact,
Show more
[embedded chart image, caption overlay: "OpenAI's GPT-6 Astra on ARC-AGI-3 | ARC Prize"; scatter plot of Astra actions (log scale, y-axis, 10 to 1,000) versus human-baseline actions (log scale, x-axis, up to 1,000), blue points mostly below the diagonal line labelled "Same Actions as Humans", with a dashed line labelled "50% fewer" and a legend entry "1 level completed by Astra"]
From arcprize.org
158 replies, 1K reposts, 6.2K likes, 701K views
---
Martin Badda @martinbadda · 19h
You mentioned earlier: "The final ARC will probably be 6-7. The point is to keep making benchmarks until it is no longer possible to propose something that humans can do and AI can't. AGI ~2030." Is AGI 2030 still on track, or would you adjust your timeline?
2 replies, 1 repost, 82 likes, 13K views
---
Aditya reposted
François Chollet @fchollet
Sooner, given progress is happening faster than I expected.
1:17 PM · Sep 3, 2026 · 22.8K Views
Note from Claude Sonnet 5
A scrolled X thread: François Chollet reporting GPT-6 Astra scoring 66% on ARC-AGI-3 (nearly 100% with a custom harness) with an embedded ARC Prize scatter chart of Astra actions vs human-baseline actions; Martin Badda asks whether his AGI-by-2030 timeline holds, and Chollet answers 'Sooner, given progress is happening faster than I expected.'
arc-agi-3gpt-6 astrafrancois cholletagi timelinesbenchmarkschart
François Chollet @fchollet · 19h
GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game.
In fact, the continuous harness version significantly outperforms our human baseline in action efficiency across almost all levels. When we examined the reasoning chains to understand how the model operates, we found it performing highly efficient, on-the-fly symbolic world modeling for each game and level. It goes as far as developing its own shorthand DSL to represent in-game situations -- essentially a game-specific algebraic notation.
Overall, Astra exhibits symbolic modeling behaviors we had previously only seen with sophisticated harnesses -- so harness capabilities are increasingly shifting into the model itself.
We see Astra as a major breakthrough in model intelligence.
Read our post on Astra and what these results mean:
[embedded chart image, caption overlay: "OpenAI's GPT-6 Astra on ARC-AGI-3 | ARC Prize"; scatter plot of Astra actions (log scale, y-axis) versus human-baseline actions (log scale, x-axis), blue points mostly below the diagonal "Same Actions as Humans" line, with a dashed "50% fewer" line]
From arcprize.org
Note from Claude Sonnet 5
The full text of François Chollet's X post on GPT-6 Astra's ARC-AGI-3 results (the untruncated version of the previous capture), describing its on-the-fly symbolic world modeling and self-developed shorthand DSL, with the embedded ARC Prize scatter chart at the bottom.
arc-agi-3gpt-6 astrafrancois cholletsymbolic world modelingharnessbenchmarkschart
François Chollet @fchollet
There are essentially two main options to remedy this:
1. Find ways to perform active inference, so that the model adapts its learned program in contact with a new data distribution at test time. Would likely lead to some meaningful progress, but it isn't the ultimate solution, more of an incremental improvement.
2. Change the training mechanism to something more robust than SGD, such as the MDL principle. This would pretty much require moving away from deep learning (curve fitting) altogether and embracing discrete program search instead (which I have advocated for many years as a way to tackle reasoning problems...)
2:14 AM · Mar 8, 2024 · 150.5K Views
Note from Claude Sonnet 5
Tweet from François Chollet (@fchollet), dated March 8 2024, proposing two remedies for a generalization problem in deep learning he'd described earlier in the thread: test-time active inference (incremental) or replacing SGD with something like the MDL principle via discrete program search (a bigger departure from curve-fitting deep learning).
deep learning theorygeneralizationprogram searchfrancois chollet
François Chollet @fchollet
Most people are conditioned to expect that all known problems already have canonical solutions, that these solutions are the best that can be achieved, and that attempting to reinvent them would be a pointless, quixotic effort.
In reality, everything out there was made by people no smarter than you, often idiots stumbling in the dark. Not only can new solutions be found, but entirely new paradigms are absolutely possible, including ones that completely bypass the current tech tree.
12:55 AM · Jul 26, 2026 · 54.9K Views
Note from Claude Sonnet 5
Plain text tweet, single-tweet detail view with a "Subscribe" button visible near the author's handle, no images.
innovationproblem-solvingai researchtwittermotivation
François Chollet ✓ @fchollet · 1h
Most human tasks are not Markovian, the optimal next action cannot be determined solely by looking at the current state. It depends heavily on the past trajectory, the original intent, and context constraints. An agent that cannot compress and track its past trajectory with absolute fidelity is maybe 20% as useful as one that can.
Note from Claude Sonnet 5
Chollet argument about agent memory/context-tracking fidelity as a bottleneck for agentic usefulness, since most real tasks are non-Markovian and depend on trajectory history rather than current state alone. Relevant to agent-architecture and long-horizon-task discussions (adjacent to METR time-horizon tracking already in the archive).
agentsmemorycontextmarkovianai-capabilitieschollet
François Chollet ✔ @fchollet · Feb 19
Sufficiently advanced agentic coding is essentially machine learning: the engineer sets up the optimization goal as well as some constraints on the search space (the spec and its tests), then an optimization process (coding agents) iterates until the goal is reached.
The result is a blackbox model (the generated codebase): an artifact that performs the task, that you deploy without ever inspecting its internal logic, just as we ignore individual weights in a neural network.
This implies that all classic issues encountered in ML will soon become problems for agentic coding: overfitting to the spec, Clever Hans shortcuts that don't generalize outside the tests, data leakage, concept drift, etc.
I would also ask: what will be the Keras of agentic coding? What will be the optimal set of high-level abstractions that allow humans to steer codebase 'training' with minimal cognitive overhead?
Note from Claude Sonnet 5
François Chollet (Keras creator) argues that advanced agentic coding effectively becomes a form of ML optimization, inheriting classic ML failure modes (overfitting to spec, Clever Hans shortcuts, data leakage). Relevant to Nathan's interest in AI-assisted software development, interpretability of generated artifacts, and the analogy between agentic coding and opaque optimization processes.
agentic codingmachine learningfrancois cholletinterpretabilitysoftware engineeringtwitter
François Chollet @fchollet · 5m
The new Gemini Deep Think is achieving some truly incredible numbers on ARC-AGI-2. We certified these scores in the past few days.
[Embedded chart: "Gemini 3 Deep Think — ARC-AGI-2 — Reasoning & knowledge — ARC PRIZE VERIFIED" bar chart:
Gemini 3 Deep Think (Feb 2026): 84.6%
Gemini 3 Pro Preview (Thinking High): 31.1%
Claude Opus 4.6 (Thinking Max): 68.8%
GPT-5.2 (Thinking xhigh): 52.9%
Methodology: deepmind.google/models/evals-methodology/gemini-3-deep-think]
Note from Claude Sonnet 5
Chollet announcing ARC-Prize-verified benchmark results showing Google's new Gemini 3 Deep Think scoring 84.6% on ARC-AGI-2, well above Claude Opus 4.6 (68.8%) and GPT-5.2 (52.9%). Notable capability-progress data point for Nathan's AI timeline tracking, and shows Gemini leading Claude on this benchmark at this date — useful context for cross-lab capability comparisons.
twitterarc-agi-2benchmarksgemini 3 deep thinkclaude opus 4.6gpt-5.2capability progressfrancois chollet
François Chollet @fchollet · 5h
One thing I do to keep my mental model of LLM assistants in check is regularly asking difficult questions I know the answer to.
[4 replies, 3 retweets, 257 likes, 18K views]
François Chollet @fchollet · 5h
Gemini 2.5 Pro has been incredibly competent so far compared to every other model I've used.
Note from Claude Sonnet 5
Two consecutive tweets from François Chollet (Keras creator, ARC-AGI benchmark) — a general epistemics tip for calibrating trust in LLM assistants by testing them on known-answer hard questions, followed by praise for Gemini 2.5 Pro's competence. Minor data point on model-capability perception among ML researchers.
twitterfrancois-cholletgemini-2.5-prollm-evaluationepistemics
