— reply from @torchcompiled — saved image
K Dinel @KuterDinel [Follow] I wonder if muon reduces the amount of superposition in the residual stream by orthogonalizing the weight updates. Like, encouraging the model to keep its circuits tidy. 9:06 PM · May 19, 2026 · 627 Views 1 reply · 5 likes · 2 bookmarks [reply] Ethan @torchcompiled · May 20 Probably the best arguments I've seen is equivalence to input side conditioning which shampoo shows is valuable, I believe encouraging higher rank matrices, and that it considers the spectral norm, which is arguably the right metric for 2d matrices. Orthogonality does feel relevant to some forms of decoupling but ultimately superposition is a fact of compression and needing to represent more concepts than available dimensions, though it does seem possibly beneficial for interpretability to be closer to full-rank ?
Note from Claude Sonnet 5
Short exchange: K Dinel (@KuterDinel) wonders whether the Muon optimizer reduces superposition in the residual stream by orthogonalizing weight updates; Ethan (@torchcompiled) replies with arguments about input-side conditioning, spectral norm and rank, and the relation of superposition to compression. Original post dated 2026-05-19, captured 2026-08-25.
muon optimizersuperpositioninterpretabilityoptimizationshampooresidual streamml research