Albert Gu @_albertgu
— quoting Hayden Prairie (@hayden_prairie)
Albert Gu @_albertgu · Apr 16
a dynamical systems point of view, which looks like an SSM applied along the residual stream, informs more principled ways to scale looped architectures
> QUOTED:
> Hayden Prairie @hayden_prairie · Apr 15
> We've been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters.
>
> Turns out the answer is YES – by looping blocks…
>
> [Diagram: recurrence block h_t → Ā, Δ, B̄ (SSM-style gating) feeding into a "Transformer Blocks" module (Multi-Head Attention + Feed Forward, residual connections) → h_t+1]
> [Two plots: "IsoFLOP Scaling Laws of Looping" — Validation Loss vs Recurrence (multiple colored curves with star-marked optima), and Optimal Recurrence vs FLOPs (log-log, roughly linear trend)]
Note from Claude Sonnet 5
Albert Gu (Mamba/SSM researcher) comments on a scaling-laws paper/thread about looping transformer blocks with an SSM-style gate applied along the residual stream, treating it as a dynamical-systems framing for how much recurrence to use at a given FLOP budget. Directly relevant to the brain_graph_1 encode/iterate-in-latent/decode architecture and its "virtualizing width via looping" primitive — a scaling-law reference for how many DEQ/loop iterations to use at different compute scales.
machine learningssmmambalooped transformersscaling lawstwitteralbert guarchitecture researchbrain_graph_1