residual stream — Timeline

2026-03-30

Martin ✓ @mjbukow It's more complex than that. Because the residual stream is purely additive, low-level gradient noise and intralayer communication signals accumulate across layers. The norm of the hidden states steadily increases with depth. In the last few layers, the model turns up the volume on output-aligned directions; effectively using the growing norm and null-space interactions to overpower accumulated noise and better align the representation for the LM head. The final RMSNorm then projects everything onto the hypersphere right before unembedding, which completely disjoints the gradient right at the end. All that to say, the gradient is going to be noisy because all of this is being transacted on one purely additive state container that serves multiple purposes in different parts of the model. Passing a clean signal back is more a prayer to AdamW than it is good ML engineering. 6:13 PM · Mar 29, 2026 · 3,109 Views

Note from Claude Sonnet 5

A technical tweet on transformer residual stream dynamics — how gradient noise accumulates across layers and how the final RMSNorm/unembedding interacts with it. Relevant to mechanistic interpretability of the residual stream, a topic adjacent to Nathan's interpretability interests.

machine learning transformers interpretability residual stream twitter gradients