Andrej Karpathy

**Andrej Karpathy** @karpathy [2025-10-01](https://x.com/karpathy/status/1973435013875314729) Finally had a chance to listen through this pod with Sutton, which was interesting and amusing. As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea > 2025-09-26 > > .@RichardSSutton, father of reinforcement learning, doesn’t think LLMs are bitter-lesson-pilled. > > My steel man of Richard’s position: we need some new architecture to enable continual (on-the-job) learning. > > And if we have continual learning, we don't need a special training --- **Andrej Karpathy** @karpathy [2025-10-02](https://x.com/karpathy/status/1973756330449236009) Hah judging by mentions overnight people seem to find the ghost analogy provocative. I swear I don't wake up just trying to come with new memes but to elaborate briefly why I thought it was a fun comparison: 1) It captures the idea that LLMs are purely digital artifacts that don't interact with the physical world (unlike animals, which are very embodied). 2) Ghosts are a kind of "echo" of the living, in this case a statistical distillation of humanity. 3) There is an air of mystery over both ghosts and LLMs, as in we don't fully understand what they are or how they work. 4) The process of training LLMs is a bit like summoning a ghost, i.e. a kind of elaborate computational ritual on a summoning platform of an exotic megastructure (GPU cluster). I've heard earlier references of LLM training as that of "summoning a demon" and it never sounded right because it implies and presupposes evil. Ghosts are a lot more neural entity just like LLMs, and may or may not be evil. For example, one of my favorite cartoons when I was a child was Casper the Friendly Ghost, clearly a friendly and wholesome entity. Same in Harry Potter, e.g. Nearly Headless Nick and such. 5) It is a nod to an earlier reference "ghost in the machine", in the context of Decartes' mind-body dualism, and of course later derived references, "Ghost in the shell" etc. As in the mind (ghost) that animates a body (machine). Probably a few other things in the embedding space. Among the ways the analogy isn't great is that while ghosts may or may not be evil, they are almost always spooky, which feels too unfair. But anyway, I like that while no analogy is perfect, they let you pull in structure laterally from one domain to another as as a way of generating entropy and reaching unique thoughts. [image: A smiling ghost-like figure made of white vapor or smoke, with two black dot eyes and a curved mouth, hovering above a computer processor chip with visible pins and circuitry on a circuit board.] --- **roon** @tszzl [2025-10-02](https://x.com/tszzl/status/1973870875758997594) “democracy of ghosts” as the model calls itself > 2025-03-11 > > we trained a new model that is good at creative writing (not sure yet how/when it will get released). this is the first time i have been really struck by something written by AI; it got the vibe of metafiction so right. > > PROMPT: > > Please write a metafictional literary short story --- **Dan Advantage** @DanAdvantage [2025-10-02](https://x.com/DanAdvantage/status/1973857944098963959) i will add to this that ghosts, like llm models, can't turn corners (they have to perform 3-point turns) --- **Swiss** @chf75 [2025-10-03](https://x.com/chf75/status/1974166392527335881) Plus, it’s the perfect time of year for this solid analogy / “accidental” meme. --- **Cosmic (t)error** @TerrorCosmic [2025-10-02](https://x.com/TerrorCosmic/status/1973898711542702527) There are Ghosts in the Shell .... --- **Scott Leibrand** @scottleibrand [2025-10-02](https://x.com/scottleibrand/status/1973768824525848715) Ghosts are spooky because they’re unfamiliar (on account of not existing IRL). In fictional worlds like Hogwarts where they are familiar, they aren’t spooky by default. LLMs’ new capabilities are spooky at first, but we quickly get used to them. --- **CottageWitchcraftCo** @the\_briarwitch [2025-10-03](https://x.com/the_briarwitch/status/1973965318675706279) traditionally ghosts tho are trapped in loops, acting out their trauma and chained to where they died, etc. They don't grow, expand in knowledge, they aren't looking forward but always back, etc. I mean I love ghosts but isn't it kind of hollow? LLMS are so full of LIFE and

**Andrej Karpathy** @karpathy 2025-09-26 Finally had a chance to listen through this pod with Sutton, which was interesting and amusing. As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea is sufficiently "bitter lesson pilled" (meaning arranged so that it benefits from added computation for free) as a proxy for whether it's going to work or worth even pursuing. The underlying assumption being that LLMs are of course highly "bitter lesson pilled" indeed, just look at LLM scaling laws where if you put compute on the x-axis, number go up and to the right. So it's amusing to see that Sutton, the author of the post, is not so sure that LLMs are "bitter lesson pilled" at all. They are trained on giant datasets of fundamentally human data, which is both 1) human generated and 2) finite. What do you do when you run out? How do you prevent a human bias? So there you have it, bitter lesson pilled LLM researchers taken down by the author of the bitter lesson - rough! In some sense, Dwarkesh (who represents the LLM researchers viewpoint in the pod) and Sutton are slightly speaking past each other because Sutton has a very different architecture in mind and LLMs break a lot of its principles. He calls himself a "classicist" and evokes the original concept of Alan Turing of building a "child machine" - a system capable of learning through experience by dynamically interacting with the world. There's no giant pretraining stage of imitating internet webpages. There's also no supervised finetuning, which he points out is absent in the animal kingdom (it's a subtle point but Sutton is right in the strong sense: animals may of course observe demonstrations, but their actions are not directly forced/"teleoperated" by other animals). Another important note he makes is that even if you just treat pretraining as an initialization of a prior before you finetune with reinforcement learning, Sutton sees the approach as tainted with human bias and fundamentally off course, a bit like when AlphaZero (which has never seen human games of Go) beats AlphaGo (which initializes from them). In Sutton's world view, all there is is an interaction with a world via reinforcement learning, where the reward functions are partially environment specific, but also intrinsically motivated, e.g. "fun", "curiosity", and related to the quality of the prediction in your world model. And the agent is always learning at test time by default, it's not trained once and then deployed thereafter. Overall, Sutton is a lot more interested in what we have common with the animal kingdom instead of what differentiates us. "If we understood a squirrel, we'd be almost done". As for my take... First, I should say that I think Sutton was a great guest for the pod and I like that the AI field maintains entropy of thought and that not everyone is exploiting the next local iteration LLMs. AI has gone through too many discrete transitions of the dominant approach to lose that. And I also think that his criticism of LLMs as not bitter lesson pilled is not inadequate. Frontier LLMs are now highly complex artifacts with a lot of humanness involved at all the stages - the foundation (the pretraining data) is all human text, the finetuning data is human and curated, the reinforcement learning environment mixture is tuned by human engineers. We do not in fact have an actual, single, clean, actually bitter lesson pilled, "turn the crank" algorithm that you could unleash upon the world and see it learn automatically from experience alone. Does such an algorithm even exist? Finding it would of course be a huge AI breakthrough. Two "example proofs" are commonly offered to argue that such a thing is possible. The first example is the success of AlphaZero learning to play Go completely from scratch with no human supervision whatsoever. But the game of Go is clearly such a simple, closed, environment that it's difficult to see the analogous formulation in the messiness of reality. I love Go, but algorithmically and categorically, it is essentially a harder version of tic tac toe. The second example is that of animals, like squirrels. And here, personally, I am also quite hesitant whether it's appropriate because animals arise by a very different computational process and via different constraints than what we have practically available to us in the industry. Animal brains are nowhere near the blank slate they appear to be at birth. First, a lot of what is commonly attributed to "learning" is imo a lot more "maturation". And second, even that which clearly is "learning" and not maturation is a lot more "finetuning" on top of something clearly powerful and preexisting. Example. A baby zebra is born and within a few dozen minutes it can run around the savannah and follow its mother. This is a highly complex sensory-motor task and there is no way in my mind that this is achieved from scratch, tabula rasa. The brains of animals and the billions of parameters within have a powerful initialization encoded in the ATCGs of their DNA, trained via the "outer loop" optimization in the course of evolution. If the baby zebra spasmed its muscles around at random as a reinforcement learning policy would have you do at initialization, it wouldn't get very far at all. Similarly, our AIs now also have neural networks with billions of parameters. These parameters need their own rich, high information density supervision signal. We are not going to re-run evolution. But we do have mountains of internet documents. Yes it is basically supervised learning that is ~absent in the animal kingdom. But it is a way to practically gather enough soft constraints over billions of parameters, to try to get to a point where you're not starting from scratch. TLDR: Pretraining is our crappy evolution. It is one candidate solution to the cold start problem, to be followed later by finetuning on tasks that look more correct, e.g. within the reinforcement learning framework, as state of the art frontier LLM labs now do pervasively. I still think it is worth to be inspired by animals. I think there are multiple powerful ideas that LLM agents are algorithmically missing that can still be adapted from animal intelligence. And I still think the bitter lesson is correct, but I see it more as something platonic to pursue, not necessarily to reach, in our real world and practically speaking. And I say both of these with double digit percent uncertainty and cheer the work of those who disagree, especially those a lot more ambitious bitter lesson wise. So that brings us to where we are. Stated plainly, today's frontier LLM research is not about building animals. It is about summoning ghosts. You can think of ghosts as a fundamentally different kind of point in the space of possible intelligences. They are muddled by humanity. Thoroughly engineered by it. They are these imperfect replicas, a kind of statistical distillation of humanity's documents with some sprinkle on top. They are not platonically bitter lesson pilled, but they are perhaps "practically" bitter lesson pilled, at least compared to a lot of what came before. It seems possibly to me that over time, we can further finetune our ghosts more and more in the direction of animals; That it's not so much a fundamental incompatibility but a matter of initialization in the intelligence space. But it's also quite possible that they diverge even further and end up permanently different, un-animal-like, but still incredibly helpful and properly world-altering. It's possible that ghosts:animals :: planes:birds. Anyway, in summary, overall and actionably, I think this pod is solid "real talk" from Sutton to the frontier LLM researchers, who might be gear shifted a little too much in the exploit mode. Probably we are still not sufficiently bitter lesson pilled and there is a very good chance of more powerful ideas and paradigms, other than exhaustive benchbuilding and benchmaxxing. And animals might be a good source of inspiration. Intrinsic motivation, fun, curiosity, empowerment, multi-agent self-play, culture. Use your imagination. > 2025-09-26 > > .@RichardSSutton, father of reinforcement learning, doesn’t think LLMs are bitter-lesson-pilled. > > My steel man of Richard’s position: we need some new architecture to enable continual (on-the-job) learning. > > And if we have continual learning, we don't need a special training --- **Andrej Karpathy** @karpathy [2025-10-01](https://x.com/karpathy/status/1973443912388977021) Something I am experimenting with. I copy pasted: 1) the full podcast transcript 2) the bitter lesson blog post 3) my full post above To ChatGPT. The interesting part is you can fork the conversation context to ask any questions and take it in whatever direction with chat: --- **renji** @brickroad7 [2025-10-01](https://x.com/brickroad7/status/1973454840635293976) You are making a colossal cultural mistake by calling them ghosts. Why unnecessarily add a spooky vibe to something that is already facing vast mountains of slander in this unbelievably important early moment where the initial conditions are set which will decide the entire long --- **Andrej Karpathy** @karpathy [2025-10-01](https://x.com/karpathy/status/1973455432359485704) Think Casper! Childhood favorite. [image: Casper flying in a cartoon scene. A white ghost with a smiling face and arms outstretched, set against a dark, spooky background with a full moon, bare trees, a haunted house, and a graveyard.] --- **Taelin** @VictorTaelin [2025-10-01](https://x.com/VictorTaelin/status/1973436226217844796) \> But the game of Go is clearly such a simple, closed, environment that it's difficult to see the analogous formulation in the messiness of reality. So perhaps a just need to find a simple, closed game where being really good at it implies being generally intelligent (: --- **gfodor.id** @gfodor [2025-10-01](https://x.com/gfodor/status/1973457751620882715) Seems like you should be able to pass the squirrel Turing test with a model that’s been hill climbed off of today’s LLMs relatively soon. Not sure what Sutton would be able to say at that point about the margin between the ideal and the reality.

**Andrej Karpathy** @karpathy [2025-07-13](https://x.com/karpathy/status/1944435412489171119) Scaling up RL is all the rage right now, I had a chat with a friend about it yesterday. I'm fairly certain RL will continue to yield more intermediate gains, but I also don't expect it to be the full story. RL is basically "hey this happened to go well (/poorly), let me slightly increase (/decrease) the probability of every action I took for the future". You get a lot more leverage from verifier functions than explicit supervision, this is great. But first, it looks suspicious asymptotically - once the tasks grow to be minutes/hours of interaction long, you're really going to do all that work just to learn a single scalar outcome at the very end, to directly weight the gradient? Beyond asymptotics and second, this doesn't feel like the human mechanism of improvement for majority of intelligence tasks. There's significantly more bits of supervision we extract per rollout via a review/reflect stage along the lines of "what went well? what didn't go so well? what should I try next time?" etc. and the lessons from this stage feel explicit, like a new string to be added to the system prompt for the future, optionally to be distilled into weights (/intuition) later a bit like sleep. In English, we say something becomes "second nature" via this process, and we're missing learning paradigms like this. The new Memory feature is maybe a primordial version of this in ChatGPT, though it is only used for customization not problem solving. Notice that there is no equivalent of this for e.g. Atari RL because there are no LLMs and no in-context learning in those domains. Example algorithm: given a task, do a few rollouts, stuff them all into one context window (along with the reward in each case), use a meta-prompt to review/reflect on what went well or not to obtain string "lesson", to be added to system prompt (or more generally modify the current lessons database). Many blanks to fill in, many tweaks possible, not obvious. Example of lesson: we know LLMs can't super easily see letters due to tokenization and can't super easily count inside the residual stream, hence 'r' in 'strawberry' being famously difficult. Claude system prompt had a "quick fix" patch - a string was added along the lines of "If the user asks you to count letters, first separate them by commas and increment an explicit counter each time and do the task like that". This string is the "lesson", explicitly instructing the model how to complete the counting task, except the question is how this might fall out from agentic practice, instead of it being hard-coded by an engineer, how can this be generalized, and how lessons can be distilled over time to not bloat context windows indefinitely. TLDR: RL will lead to more gains because when done well, it is a lot more leveraged, bitter-lesson-pilled, and superior to SFT. It doesn't feel like the full story, especially as rollout lengths continue to expand. There are more S curves to find beyond, possibly specific to LLMs and without analogues in game/robotics-like environments, which is exciting. --- **Yuchen Jin** @Yuchenj\_UW [2025-07-13](https://x.com/Yuchenj_UW/status/1944438386153939221) Very insightful. The new training paradigm (can be called “lesson-based learning”) can be a self-supervised memory + retrieval + reflection feedback loop, without any external reward signal. who is building this? --- **Noam Brown** @polynoamial [2025-07-13](https://x.com/polynoamial/status/1944443577553289281) Indeed, there is still more research to be done --- **Danielle Fong ** @DanielleFong [2025-07-13](https://x.com/DanielleFong/status/1944436669257765260) \> The new Memory feature is maybe a primordial version of this in ChatGPT, though it is only used for customization not problem solving. Notice that there is no equivalent of this for e.g. Atari RL because there are no LLMs and no in-context learning in those domains. it's not --- **Andrew Zhao@ICML25** @\_AndrewZhao [2025-07-13](https://x.com/_AndrewZhao/status/1944439528594268328) It’s been built and expanded since 2023 https://arxiv.org/abs/2308.10144 --- **tokenbender** @tokenbender [2025-07-13](https://x.com/tokenbender/status/1944453139261988948) the key lies in not discarding everything that could possibly be useful. active response filtering, algorithms that are naturally entropy/exploration friendly, repeatedly listening and learning from the good samples. --- **Christoph Friedler** @ChrisFriedler [2025-07-13](https://x.com/ChrisFriedler/status/1944438051511337292) The goal will be to have everything set up like @GoogleDeepMind’s AlphaEvolve: iterating with genetic algorithms on multiple solutions, saving promising results to a database, using AI as a judge, retrieving promising solutions when stuck, and applying step functions to track --- **elvis** @omarsar0 [2025-07-13](https://x.com/omarsar0/status/1944444433371414860) Yeah, when I read all these RL-driven approaches for LLMs, it does feel like we are just scratching the surface.

Andrej Karpat... @karpat... · Feb 24 Agency > Intelligence I had this intuitively wrong for decades, I think due to a pervasive cultural veneration of intelligence, various entertainment/media, obsession with IQ etc. Agency is significantly more powerful and significantly more scarce. Are you hiring for agency? Are [Show more] > QUOTED: Garry Tan @garrytan · Feb 24 > Intelligence is on tap now so agency is even more important x.com/hvpandya/statu... [734 replies, 3.7K reposts, 19K likes, 1.5M views] Noam Brown @polynoamial · 2h Do you really think AI models won't have agency soon too? [29 replies, 9 reposts, 180 likes, 14K views]

Andrej Karpathy @karpathy You can tell the RL is done properly when the models cease to speak English in their chain of thought 11:10 PM · Sep 15, 2024 · 371.5K Views 273 Reposts 45 Quotes 4,888 Likes 680 Bookmarks