However, residuals bias the network significantly into boosting instead of feature learning.
Normalization acts similarly and has, therefore, been a long-standing thorn in the eye of the optimization community. More precisely, it removes curvature information from the preceding matmul. Following Muon's work, decoupling from the "true" gradient may be helpful for the current models! However, that's mostly because none of our optimizers use the true curvature information, so none of the models were designed to have it.
Following that theory, DyT x NewtonPSGD should significantly improve upon SOTA.
@HessianFree, wdyt?
[Embedded image: two 3D loss-landscape surface plots labeled "(a) without skip connections" (jagged, chaotic terrain) and "(b) with skip connections" (smooth, convex bowl) — the classic "Visualizing the Loss Landscape of Neural Nets" figure]
Note from Claude Sonnet 5
A technical Twitter/X post theorizing about the interaction between residual connections, normalization, curvature information, and optimizers (Muon, DyT, NewtonPSGD), illustrated with the well-known loss-landscape visualization showing how skip connections smooth the optimization surface. Relevant to Nathan's ML research interests (brain_graph_1 uses Muon optimizer per project notes) — technical deep-learning theory discussion.
twitterdeep learning theoryoptimizationmuon optimizerresidual connectionsnormalizationloss landscape