— saved image
Soufiane Hayou @hayou_soufiane · 1h It's easy to tell whether Muon was used to train a model just by looking at the empirical spectral distribution of its hidden layer weights. Compared with SGD/Adam, Muon leaves a very distinctive spectral signature in W^T W. This is because Muon updates low-magnitude directions more aggressively. [embedded chart: histogram titled "Empirical spectral density, layer net.4 (pooled over seeds)"; x-axis "log10(eigenvalue of W^T W)" from about -8 to 3; y-axis "density" 0.0 to 0.7; legend sgd (blue), adam (orange), muon (green)] 2 replies · 22 likes · 1K views Xidulu @xidulu · 1h This is also noted in Kimi's "Muon is Scalable for LLM Training" paper... but I actually wonder what's the downstream sequences are: Which type of spectrum learns / forgets more / less [embedded figure: six small line plots (AttnQO, AttnKV, Experts, SharedExperts, Router, Dense), y-axis "SVD Entropy", x-axis "Training Iterations (K)", legend AdamW (red) and Muon (blue)] Figure 4: SVD entropy of weight matrices across different training iterations. We categorize the weight matrices into 6 different groups: 1) AttnQO denotes the weight matrices related to the query and output projection in the attention layer; 2) AttnKV denotes the weight matrices related to the key and value projection in the attention layer; 3) Experts denotes the weight matrices in expert models; 4) SharedExperts denotes the weight matrices in shared expert models; 5) Router denotes the weight matrices in the router; 6) Dense denotes the weight matrices in the first dense layer. The SVD entropy is calculated as the macro-average of the weight matrices in each group across all layers. For weights in expert models, we only calculate 3 out of 64 experts in different layers for efficiency.
Note from Claude Sonnet 5
Thread on the spectral signature of the Muon optimizer: Soufiane Hayou says Muon-trained weights have a distinctive W^T W spectrum versus SGD/Adam (histogram shows Muon's eigenvalue density shifted to high log-eigenvalues, SGD/Adam peaked near 0), and Xidulu replies citing Kimi's 'Muon is Scalable for LLM Training' SVD-entropy figure and wondering about downstream learning/forgetting effects.
muon optimizerspectral analysisweight spectrumadamsgdkimisvd entropyml training