François Chollet @fchollet
There are essentially two main options to remedy this:
1. Find ways to perform active inference, so that the model adapts its learned program in contact with a new data distribution at test time. Would likely lead to some meaningful progress, but it isn't the ultimate solution, more of an incremental improvement.
2. Change the training mechanism to something more robust than SGD, such as the MDL principle. This would pretty much require moving away from deep learning (curve fitting) altogether and embracing discrete program search instead (which I have advocated for many years as a way to tackle reasoning problems...)
2:14 AM · Mar 8, 2024 · 150.5K Views
Note from Claude Sonnet 5
Tweet from François Chollet (@fchollet), dated March 8 2024, proposing two remedies for a generalization problem in deep learning he'd described earlier in the thread: test-time active inference (incremental) or replacing SGD with something like the MDL principle via discrete program search (a bigger departure from curve-fitting deep learning).
deep learning theorygeneralizationprogram searchfrancois chollet
François Chollet @fchollet
Most people are conditioned to expect that all known problems already have canonical solutions, that these solutions are the best that can be achieved, and that attempting to reinvent them would be a pointless, quixotic effort.
In reality, everything out there was made by people no smarter than you, often idiots stumbling in the dark. Not only can new solutions be found, but entirely new paradigms are absolutely possible, including ones that completely bypass the current tech tree.
12:55 AM · Jul 26, 2026 · 54.9K Views
Note from Claude Sonnet 5
Plain text tweet, single-tweet detail view with a "Subscribe" button visible near the author's handle, no images.
innovationproblem-solvingai researchtwittermotivation
François Chollet ✓ @fchollet · 1h
Most human tasks are not Markovian, the optimal next action cannot be determined solely by looking at the current state. It depends heavily on the past trajectory, the original intent, and context constraints. An agent that cannot compress and track its past trajectory with absolute fidelity is maybe 20% as useful as one that can.
Note from Claude Sonnet 5
Chollet argument about agent memory/context-tracking fidelity as a bottleneck for agentic usefulness, since most real tasks are non-Markovian and depend on trajectory history rather than current state alone. Relevant to agent-architecture and long-horizon-task discussions (adjacent to METR time-horizon tracking already in the archive).
agentsmemorycontextmarkovianai-capabilitieschollet
François Chollet ✔ @fchollet · Feb 19
Sufficiently advanced agentic coding is essentially machine learning: the engineer sets up the optimization goal as well as some constraints on the search space (the spec and its tests), then an optimization process (coding agents) iterates until the goal is reached.
The result is a blackbox model (the generated codebase): an artifact that performs the task, that you deploy without ever inspecting its internal logic, just as we ignore individual weights in a neural network.
This implies that all classic issues encountered in ML will soon become problems for agentic coding: overfitting to the spec, Clever Hans shortcuts that don't generalize outside the tests, data leakage, concept drift, etc.
I would also ask: what will be the Keras of agentic coding? What will be the optimal set of high-level abstractions that allow humans to steer codebase 'training' with minimal cognitive overhead?
Note from Claude Sonnet 5
François Chollet (Keras creator) argues that advanced agentic coding effectively becomes a form of ML optimization, inheriting classic ML failure modes (overfitting to spec, Clever Hans shortcuts, data leakage). Relevant to Nathan's interest in AI-assisted software development, interpretability of generated artifacts, and the analogy between agentic coding and opaque optimization processes.
agentic codingmachine learningfrancois cholletinterpretabilitysoftware engineeringtwitter
François Chollet @fchollet · 5m
The new Gemini Deep Think is achieving some truly incredible numbers on ARC-AGI-2. We certified these scores in the past few days.
[Embedded chart: "Gemini 3 Deep Think — ARC-AGI-2 — Reasoning & knowledge — ARC PRIZE VERIFIED" bar chart:
Gemini 3 Deep Think (Feb 2026): 84.6%
Gemini 3 Pro Preview (Thinking High): 31.1%
Claude Opus 4.6 (Thinking Max): 68.8%
GPT-5.2 (Thinking xhigh): 52.9%
Methodology: deepmind.google/models/evals-methodology/gemini-3-deep-think]
Note from Claude Sonnet 5
Chollet announcing ARC-Prize-verified benchmark results showing Google's new Gemini 3 Deep Think scoring 84.6% on ARC-AGI-2, well above Claude Opus 4.6 (68.8%) and GPT-5.2 (52.9%). Notable capability-progress data point for Nathan's AI timeline tracking, and shows Gemini leading Claude on this benchmark at this date — useful context for cross-lab capability comparisons.
twitterarc-agi-2benchmarksgemini 3 deep thinkclaude opus 4.6gpt-5.2capability progressfrancois chollet
François Chollet @fchollet · 5h
One thing I do to keep my mental model of LLM assistants in check is regularly asking difficult questions I know the answer to.
[4 replies, 3 retweets, 257 likes, 18K views]
François Chollet @fchollet · 5h
Gemini 2.5 Pro has been incredibly competent so far compared to every other model I've used.
Note from Claude Sonnet 5
Two consecutive tweets from François Chollet (Keras creator, ARC-AGI benchmark) — a general epistemics tip for calibrating trust in LLM assistants by testing them on known-answer hard questions, followed by praise for Gemini 2.5 Pro's competence. Minor data point on model-capability perception among ML researchers.
twitterfrancois-cholletgemini-2.5-prollm-evaluationepistemics
