← All topics

sgd

4 captures, most recent first.

@hayou_soufiane

— saved image

Soufiane Hayou @hayou_soufiane · 1h
It's easy to tell whether Muon was used to train a model just by looking at the empirical spectral distribution of its hidden layer weights. Compared with SGD/Adam, Muon leaves a very distinctive spectral signature in W^T W. This is because Muon updates low-magnitude directions more aggressively.

[embedded chart: histogram titled "Empirical spectral density, layer net.4 (pooled over seeds)"; x-axis "log10(eigenvalue of W^T W)" from about -8 to 3; y-axis "density" 0.0 to 0.7; legend sgd (blue), adam (orange), muon (green)]

2 replies · 22 likes · 1K views

Xidulu @xidulu · 1h
This is also noted in Kimi's "Muon is Scalable for LLM Training" paper... but I actually wonder what's the downstream sequences are: Which type of spectrum learns / forgets more / less

[embedded figure: six small line plots (AttnQO, AttnKV, Experts, SharedExperts, Router, Dense), y-axis "SVD Entropy", x-axis "Training Iterations (K)", legend AdamW (red) and Muon (blue)]
Figure 4: SVD entropy of weight matrices across different training iterations. We categorize the weight matrices into 6 different groups: 1) AttnQO denotes the weight matrices related to the query and output projection in the attention layer; 2) AttnKV denotes the weight matrices related to the key and value projection in the attention layer; 3) Experts denotes the weight matrices in expert models; 4) SharedExperts denotes the weight matrices in shared expert models; 5) Router denotes the weight matrices in the router; 6) Dense denotes the weight matrices in the first dense layer. The SVD entropy is calculated as the macro-average of the weight matrices in each group across all layers. For weights in expert models, we only calculate 3 out of 64 experts in different layers for efficiency.
Note from Claude Sonnet 5

Thread on the spectral signature of the Muon optimizer: Soufiane Hayou says Muon-trained weights have a distinctive W^T W spectrum versus SGD/Adam (histogram shows Muon's eigenvalue density shifted to high log-eigenvalues, SGD/Adam peaked near 0), and Xidulu replies citing Kimi's 'Muon is Scalable for LLM Training' SVD-entropy figure and wondering about downstream learning/forgetting effects.

muon optimizerspectral analysisweight spectrumadamsgdkimisvd entropyml training

Dimitris Papailiopoulos @DimitrisPapail

Dimitris Papailiopo... (@DimitrisPa... ✓) — 1h btw this is a weird heavy ball SGD variant that basically does this 1. Load previous checkpoint weights. 2. Reset optimizer state / momentum buffers. 3. Train N steps. 4. For first M steps: warm LR from 0.1x -> 1.0x. 5. Hold LR flat until ~50% of the cycle. 6. Linearly decay LR to zero. 7. Save checkpoint. 8. Repeat the optimizer is exactly this buf = mu * buf + grad p *= 1 - lr * wd p -= lr * buf / 524288 @CevherLIONS @_arohan_ does this have a name? I'd call it wave SGD lol
Note from Claude Sonnet 5

Text-only tweet with pseudocode for an optimizer variant; no images.

machine learningoptimizersgdtwittertechnical

Kevin Frans @kvfrans

— quoting @micahgold... (Micah Gold...)

Kevin Frans ✅ @kvfrans · 16h This is a great investigation, and it answers the itch that often comes up when using gradient accumulation -- why not just make more frequent updates? The answer is that you need to properly scale the b1/b2 settings too! > QUOTED: Micah Gold... @micahgold... · 20h > 🚨 Did you know that small-batch vanilla SGD without momentum (i.e. the first optimizer you learn about in intro ML) is virtually as fast as AdamW for LLM pretraining on a per-FLOP basis? ... > [Line chart titled "GPT-3 (1.3B)": FineWeb loss vs training tokens (0–10B) for two curves — AdamW BS=512 (blue) and SGD no momentum BS=1 (orange). Both curves track closely together, descending from ~4.3 to ~2.9-3.0 loss, with orange showing a few large early spikes.]
Note from Claude Sonnet 5

A machine-learning Twitter exchange about optimizer scaling laws — the surprising finding that small-batch vanilla SGD without momentum tracks AdamW's per-FLOP training efficiency for LLM pretraining, plus a note on scaling beta1/beta2 with gradient accumulation. Technical reading consistent with Nathan's interest in optimizer behavior for his own model training.

machine learningoptimizerssgdadamwllm pretrainingtwittertechnical discussion

Vlado Boza @bozavlado

— reply from Lucas Beyer (bl16) (@giffmana)

Vlado Boza @bozavlado · 2h If you do pure SGD without momentum, you can get away with 0 memory overhead (but you need 32bit weights), just apply update during be pass (lomo/adalomo does it), here is the main trick pytorch.org/docs/stable/ge... 💬2 ♡6 📊221 Lucas Beyer (bl16) @giffmana · 2h yeah exactly that's the big pro of pure SGD! Thanks for the pointer to the hook.
Note from Claude Sonnet 5

Technical ML training discussion about memory-efficient SGD optimization (LOMO/AdaLoMo technique, applying gradient updates during the backward pass to avoid storing gradients). Not AI-safety focused, general ML training tip.

machine-learningtrainingoptimizationsgdmemory-efficiencytechnical