bitter lesson

j⧉nus @repligate

2026-08-19 — reply chain (quoted tweet, author unlabeled in this crop) and @repligate — saved image

[cut off at top] look up facts or redo a mathematical derivation all the time there is a higher chance of mistakes, which can compound in a long-horizon task.

Once you buy that it is valuable to do things parametrically without tool use, then you must buy the argument that a 1B cognitive core is not sufficient. There is an information limit to how much knowledge can be internalized by a 1B model, and we will surely want AI to know more than that. Even 1T probably won't be enough. We will want the AI to know as much about our world as possible, we will want it to be updated with new information, and our expectations of what AI can do for us will continue to grow.

In summary, tool use enables small models to do a lot more, but those who demand the highest quality intelligence will always want larger models. Bitter lesson strikes again.

73 replies, 153 reposts, 1K likes, 268K views

j⧉nus @repligate
but it also seems like larger models have a stronger cognitive core, not just more world knowledge - in the sense of being able to integrate new information as well as grok fundamentals better, and this seems to have continued to improve even going from opus to fable.

3:16 AM · Aug 19, 2026 · 3,527 Views

ai cognitioncognitive corebitter lessonmodel scalingclaudeopusfablejanustwitter discourse

[cut off at top, continuing from previous screenshot] In the same way, language models knowing a fact internally, without tool calls, is meaningful. The first reason is that we obviously care about speed; you'd much rather get an answer immediately than have the model think a long time to be sure of its answer or browse the web. A second reason is that there are some things that are simply best learned via backpropagation over lots of data. If you ask about how people generally think of the Shambhala music festival, you'd rather a large language model give you an aggregate opinion based on all the data on the internet, than get a regurgitation of the first three reviews that show up in a web search. A third reason is that having to do a lot of work to find an answer is not as reliable as already knowing the answer. While this does not have to be true in theory, it is probably true in practice, at least for now. If you have to re-look up facts or redo a mathematical derivation all the time there is a higher chance of mistakes, which can compound in a long-horizon task.

Once you buy that it is valuable to do things parametrically without tool use, then you must buy the argument that a 1B cognitive core is not sufficient. There is an information limit to how much knowledge can be internalized by a 1B model, and we will surely want AI to know more than that. Even 1T probably won't be enough. We will want the AI to know as much about our world as possible, we will want it to be updated with new information, and our expectations of what AI can do for us will continue to grow.

In summary, tool use enables small models to do a lot more, but those who demand the highest quality intelligence will always want larger models. Bitter lesson strikes again.

73 replies, 153 reposts, 1K likes, 268K views

ai cognitioncognitive coretool usemodel scalingjason weibitter lessontwitter discourse