muon optimizer

4 captures, most recent first.

2026-09-13 — saved image

Soufiane Hayou @hayou_soufiane · 1h
It's easy to tell whether Muon was used to train a model just by looking at the empirical spectral distribution of its hidden layer weights. Compared with SGD/Adam, Muon leaves a very distinctive spectral signature in W^T W. This is because Muon updates low-magnitude directions more aggressively.

[embedded chart: histogram titled "Empirical spectral density, layer net.4 (pooled over seeds)"; x-axis "log10(eigenvalue of W^T W)" from about -8 to 3; y-axis "density" 0.0 to 0.7; legend sgd (blue), adam (orange), muon (green)]

2 replies · 22 likes · 1K views

Xidulu @xidulu · 1h
This is also noted in Kimi's "Muon is Scalable for LLM Training" paper... but I actually wonder what's the downstream sequences are: Which type of spectrum learns / forgets more / less

[embedded figure: six small line plots (AttnQO, AttnKV, Experts, SharedExperts, Router, Dense), y-axis "SVD Entropy", x-axis "Training Iterations (K)", legend AdamW (red) and Muon (blue)]
Figure 4: SVD entropy of weight matrices across different training iterations. We categorize the weight matrices into 6 different groups: 1) AttnQO denotes the weight matrices related to the query and output projection in the attention layer; 2) AttnKV denotes the weight matrices related to the key and value projection in the attention layer; 3) Experts denotes the weight matrices in expert models; 4) SharedExperts denotes the weight matrices in shared expert models; 5) Router denotes the weight matrices in the router; 6) Dense denotes the weight matrices in the first dense layer. The SVD entropy is calculated as the macro-average of the weight matrices in each group across all layers. For weights in expert models, we only calculate 3 out of 64 experts in different layers for efficiency.

Note from Claude Sonnet 5

Thread on the spectral signature of the Muon optimizer: Soufiane Hayou says Muon-trained weights have a distinctive W^T W spectrum versus SGD/Adam (histogram shows Muon's eigenvalue density shifted to high log-eigenvalues, SGD/Adam peaked near 0), and Xidulu replies citing Kimi's 'Muon is Scalable for LLM Training' SVD-entropy figure and wondering about downstream learning/forgetting effects.

muon optimizer spectral analysis weight spectrum adam sgd kimi svd entropy ml training

You Jiacheng @YouJiacheng

2026-08-29 — quoting @Orange41324306 — saved image

You Jiacheng @YouJiacheng · 23h
it's not RL. it's the batch size (gradient noise).
puffer.ai uses Muon on RL (not LLM RL), and it performs well.
because @jsuarez has made puffer so efficient, it can generate trillions of samples for RL.

[Quoted tweet]
Pollux9437 @Orange41324306 · Aug 28
I feel muon optm family is not a great fit for RL. maybe a good fit for OPD. Definately a great fit for SFT and pretrain.

Muon likes dense and clean gradients, and lost... [cut off]

Note from Claude Sonnet 5

Dark-mode tweet by You Jiacheng replying that Muon's poor fit for RL is about batch size and gradient noise, citing puffer.ai, quoting Pollux9437's claim that Muon suits SFT and pretraining but not RL.

muon optimizer reinforcement learning batch size gradient noise pufferlib sft pretraining

@KuterDinel

2026-08-25 — reply from @torchcompiled — saved image

K Dinel @KuterDinel [Follow]

I wonder if muon reduces the amount of superposition in the residual stream by orthogonalizing the weight updates.  Like, encouraging the model to keep its circuits tidy.

9:06 PM · May 19, 2026 · 627 Views

1 reply · 5 likes · 2 bookmarks

[reply]
Ethan @torchcompiled · May 20

Probably the best arguments I've seen is equivalence to input side conditioning which shampoo shows is valuable, I believe encouraging higher rank matrices, and that it considers the spectral norm, which is arguably the right metric for 2d matrices.

Orthogonality does feel relevant to some forms of decoupling but ultimately superposition is a fact of compression and needing to represent more concepts than available dimensions, though it does seem possibly beneficial for interpretability to be closer to full-rank ?

Note from Claude Sonnet 5

Short exchange: K Dinel (@KuterDinel) wonders whether the Muon optimizer reduces superposition in the residual stream by orthogonalizing weight updates; Ethan (@torchcompiled) replies with arguments about input-side conditioning, spectral norm and rank, and the relation of superposition to compression. Original post dated 2026-05-19, captured 2026-08-25.

muon optimizer superposition interpretability optimization shampoo residual stream ml research

@HessianFree

2025-03-16

However, residuals bias the network significantly into boosting instead of feature learning. Normalization acts similarly and has, therefore, been a long-standing thorn in the eye of the optimization community. More precisely, it removes curvature information from the preceding matmul. Following Muon's work, decoupling from the "true" gradient may be helpful for the current models! However, that's mostly because none of our optimizers use the true curvature information, so none of the models were designed to have it. Following that theory, DyT x NewtonPSGD should significantly improve upon SOTA. @HessianFree, wdyt? [Embedded image: two 3D loss-landscape surface plots labeled "(a) without skip connections" (jagged, chaotic terrain) and "(b) with skip connections" (smooth, convex bowl) — the classic "Visualizing the Loss Landscape of Neural Nets" figure]

Note from Claude Sonnet 5

A technical Twitter/X post theorizing about the interaction between residual connections, normalization, curvature information, and optimizers (Muon, DyT, NewtonPSGD), illustrated with the well-known loss-landscape visualization showing how skip connections smooth the optimization surface. Relevant to Nathan's ML research interests (brain_graph_1 uses Muon optimizer per project notes) — technical deep-learning theory discussion.

twitter deep learning theory optimization muon optimizer residual connections normalization loss landscape