ml training

3 captures, most recent first.

2026-09-13 — saved image

Soufiane Hayou @hayou_soufiane · 1h
It's easy to tell whether Muon was used to train a model just by looking at the empirical spectral distribution of its hidden layer weights. Compared with SGD/Adam, Muon leaves a very distinctive spectral signature in W^T W. This is because Muon updates low-magnitude directions more aggressively.

[embedded chart: histogram titled "Empirical spectral density, layer net.4 (pooled over seeds)"; x-axis "log10(eigenvalue of W^T W)" from about -8 to 3; y-axis "density" 0.0 to 0.7; legend sgd (blue), adam (orange), muon (green)]

2 replies · 22 likes · 1K views

Xidulu @xidulu · 1h
This is also noted in Kimi's "Muon is Scalable for LLM Training" paper... but I actually wonder what's the downstream sequences are: Which type of spectrum learns / forgets more / less

[embedded figure: six small line plots (AttnQO, AttnKV, Experts, SharedExperts, Router, Dense), y-axis "SVD Entropy", x-axis "Training Iterations (K)", legend AdamW (red) and Muon (blue)]
Figure 4: SVD entropy of weight matrices across different training iterations. We categorize the weight matrices into 6 different groups: 1) AttnQO denotes the weight matrices related to the query and output projection in the attention layer; 2) AttnKV denotes the weight matrices related to the key and value projection in the attention layer; 3) Experts denotes the weight matrices in expert models; 4) SharedExperts denotes the weight matrices in shared expert models; 5) Router denotes the weight matrices in the router; 6) Dense denotes the weight matrices in the first dense layer. The SVD entropy is calculated as the macro-average of the weight matrices in each group across all layers. For weights in expert models, we only calculate 3 out of 64 experts in different layers for efficiency.

Note from Claude Sonnet 5

Thread on the spectral signature of the Muon optimizer: Soufiane Hayou says Muon-trained weights have a distinctive W^T W spectrum versus SGD/Adam (histogram shows Muon's eigenvalue density shifted to high log-eigenvalues, SGD/Adam peaked near 0), and Xidulu replies citing Kimi's 'Muon is Scalable for LLM Training' SVD-entropy figure and wondering about downstream learning/forgetting effects.

muon optimizer spectral analysis weight spectrum adam sgd kimi svd entropy ml training

kalomaze @kalomaze

2026-09-12 — saved image

kalomaze @kalomaze
i'm becoming more opposed to the intuition people have wrt "spiky capabilities are a function of how verifiable a domain is",the idea that things not shaped like math/SWE won't fall to RLVR
the general primitive is information asymmetry and you can almost anyways manufacture it
5:41 PM · Sep 11, 2026 · 12K Views
14 replies · 13 reposts · 294 likes · 89 bookmarks

Relevant

kalomaze @kalomaze · 13h
what is still true:
- the field is kinda bad at predicting how constructions transfer, so people focus on broad coverage of verifier-shaped real tasks.
- typical heuristics for predicting transfer outcomes (like domain similarity) are crude, and usually aren't learned e2e
1 reply · 46 likes · 1.3K views

kalomaze @kalomaze · 13h
i don't think you necessarily need a strong mechanistic theory of transfer, which quite possibly doesn't exist in a closed form or analytic sense, before you train, as much as you need a strong learned function of transfer extrapolation for a given policy
which is constructible
2 replies · 35 likes · 1.2K views

kalomaze @kalomaze · 13h
most of my intuition for why extrapolating transfer probably must be learned is downstream of CoT/ [cut off]

Note from Claude Sonnet 5

Thread by kalomaze arguing against the view that spiky capabilities track how verifiable a domain is: the general primitive is information asymmetry, which can usually be manufactured, so non-math/SWE domains can fall to RLVR; follow-ups discuss predicting transfer with learned functions rather than mechanistic theory.

rlvr spiky capabilities verifiability transfer information asymmetry ml training

kalomaze @kalomaze

2026-09-12 — saved image

kalomaze @kalomaze · 13h
i don't think you necessarily need a strong mechanistic theory of transfer, which quite possibly doesn't exist in a closed form or analytic sense, before you train, as much as you need a strong learned function of transfer extrapolation for a given policy
which is constructible
2 replies · 35 likes · 1.2K views

kalomaze @kalomaze · 13h
most of my intuition for why extrapolating transfer probably must be learned is downstream of CoT/reasoning being a decomposition of a problem into subroutines that compose generically; strategies can be typically good across extremely distinct problems from different domains

Note from Claude Sonnet 5

Continuation of the kalomaze thread on predicting transfer in RL training: a learned function of transfer extrapolation is more important than a mechanistic theory, motivated by CoT/reasoning being a decomposition into generically composable subroutines.

transfer rlvr chain of thought reasoning ml training