gsm8k

Omar Khattab ✓ @lateinteraction · 22h Buried in the massive progress in LLMs over the past few years is how all your favorite Transformers/DNNs still can't solve even just grade school math problems above a "B" grade through a forward pass. Unless they're in a scaffold like CoT, ReAct, RLM, etc. And this is true even at trillions of params and bajillions of FLOPs. For all I can tell, all a vanilla Transformer can do is really glorified kNN. Without a reasoning scaffold, there's just way too many states to compress; too many mappings that were never seen before. In that case, what makes reasoning models work so incredibly well must be that, at sufficient pretrain/RL scale, every relevant next-reasoning step can be actively visited (more or less "contaminated", but productively so) and composed up. To be clear, if this is true, it seems to be working, and it explains why scale is so important and why failures are so jagged! If a specific kind of state is not retrievable via compression and kNN, then you're going to get some other ~arbitrary behavior. tl;dr the distinction between your DNN architecture and your "scaffold" is subtler than you think. > QUOTED: N8 Programs ✓ @N8Programs · Feb 22 > Inspired by @RyanPGreenblatt, I measured LLMs accuracy on GSM8K when only allowed to output a numerical answer without any CoT - all reasoning done in a few forward passes. The result is a nice log-linear scaling curve. We can ... > [Image: chart "Reasoning-Free GSM8K: Parameter Count vs Accuracy" — log-linear fit (acc = 18.5*log10(p) + 11.5, R²=0.877), scatter of model points including Qwen3.5-297B, Sonnet 4.6, GPT-5.2, Kimi K2.5]