7 captures, most recent first.
Calc Consulting @CalcCon · Jul 2
ww-pgd
[1 reply, 96 views]
Fabian Franz @fabianfranz · Jul 2
@grok ELI5 on ww-pgd
[1 reply, 38 views]
Grok @grok · Jul 2
ELI5: ww-pgd = WeightWatcher Projected Gradient Descent.
It's a PyTorch add-on that wraps normal optimizers (AdamW etc.).
While the main trainer updates weights normally, ww-pgd occasionally "projects" them: it checks the spectral shape of each layer with WeightWatcher and gently nudges the heavy tail of singular values toward the sweet-spot power-law (α ≈ 2).
Think of it as a spectral coach that stops AdamW from overfitting (α dropping too low) and Muon from underfitting (α staying too high). Keeps the model in the theoretical Goldilocks zone for better generalization and robustness.
Note from Claude Sonnet 5
Plain text reply thread, no images; technical ML explainer from Grok.
twittermachine-learningoptimizersweightwatchertechnical
א-e/acc (@murage_kibicho) — 17h
The interesting thing I heard: big labs don't use Muon or AdamW during the million dollar training runs.
They all finetune SGD with a spectral norm regularizer and it saves them the extra mean and variance bytes. ie. larger batches. I wonder if it's true.
> QUOTED: hallerite (@hallerite) — Jun 28
> talking to people at the big labs is so boring because they will never tell you anything interesting. really grateful to work at a company that not only allows me to talk about the things I work on, but even encourages me to tweet mor...
Note from Claude Sonnet 5
Tweet speculating about optimizer choices at large AI labs, quoting an unrelated tweet about lab secrecy culture.
machine learningoptimizersai labstwitter
Ji-Ha @Ji_Ha_Kim · 9h
Very cool! I worked on this recently, and I actually used an identical approach early on. But I believe there is a significantly better approach – a **single** minimax rational iteration can beat 5 polynomial steps!
[Embedded image: math graphing tool (Desmos-like) showing equations φ(x)=φ2(φ1(x)), Z_7(x)=Z_1(x)/1.4093601263725774, p(x)=p5(p4(p3(p2(p1(x))))), and various polynomial definitions φ1(x), φ2(x), p1(x)-p5(x), plotted as oscillating curves converging near y=1]
> QUOTED: Jack Zhang ✓ @jcz42 · Mar 30
> We made Muon run up to 2x faster for free!
> Introducing Gram Newton-Schulz: a mathematically equivalent but ...
> [thumbnail bar chart comparing "1 Dense MLP Weight" pipeline stages, showing speedup]
Note from Claude Sonnet 5
Technical ML optimization thread — a reply proposing a minimax rational iteration approach to speed up Newton-Schulz iteration (used for the Muon optimizer's orthogonalization step), replying to a "Gram Newton-Schulz" 2x speedup announcement. Continues the same optimizer/ML-engineering thread cluster as other screenshots in this batch.
machine learningoptimizersmuontwitternumerical methods
Harsh Bhatt ✓ @harshbhatt7585 · 23h
I tried moun over AdamW with GRPO, it is far stable than that with moun.
[2 replies, 371 views]
rajan agarwal ✓ @_rajanagarwal · 23h
my intuition here is that muon probably provides strong benefits when the intention is generalized RL, but there's a chance that only works with muon bases just by the argument of loss landscapes
[1 reply, 321 views]
Harsh Bhatt ✓ @harshbhatt7585 · 22h
I tried in high dense grid world with HRM model, In my GRPO experiments, Muon felt noticeably more stable than AdamW. I also think that operator-aware step geometry may help more when the RL objective is broad and general rather than narrowly optimized, But grid-world was very narrowed as cannot be as general as LLMs but still it felt better, let me search the wandb experiment, will drop here if I find.
[1 reply, 2 likes, 95 views]
Harsh Bhatt ✓ @harshbhatt7585 · 22h
*HRM model wa not pretrained LLM
[96 views]
Mingta Kaivo 明塔 ... ✓ @MingtaKa... · 12h
super interesting thread. we've been experimenting with different optimizers for fine-tuning audio models at audiowave and the step size sensitivity is real. curious if anyone has tried muon specifically for RLHF on smaller models (sub-7B) or if the benefits mostly show up at scale?
Note from Claude Sonnet 5
An ML researcher discussion thread comparing the Muon optimizer vs AdamW for GRPO (a policy-gradient RL algorithm used in RLHF-style training), including stability observations across grid-world and audio model fine-tuning experiments. Technical ML-training content.
machine learningoptimizersmuonadamwgrporlhftwitterreinforcement learning
Cameron R. Wolf... ✓ @cwolfere... · Jan 14
Very rare to see useful modifications to optimizers these days. AdamW has dominated for so long with LLMs. Cautious weight decay (CWD) is a pretty simple (one-line) change to decoupled weight decay that consistently improves performance.
From the paper: "Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer update."
[Quoted tweet:]
varun ✓ @varunneal · Jan 14
Cautious Weight Decay is a surprisingly simple technique that has been repeatedly validated in Modded NanoGPT. I expect it will gain serious traction as the default variant of decoupled weight decay
x.com/Tim38463182/st...
[Embedded code snippet:]
apply_update(param, update, learning_rate, weight_decay):
mask = (update * param) >= 0
update += weight_decay * param * mask
return param - learning_rate * update
Note from Claude Sonnet 5
A technical ML tweet describing "Cautious Weight Decay" (CWD), a simple one-line modification to AdamW-style decoupled weight decay that only applies decay where the sign matches the optimizer update, validated in the Modded NanoGPT speedrun community. Relevant to Nathan's own ML/optimizer engineering work on brain_graph_1 (which is already tracking Muon and other optimizer choices per project memory) — a candidate technique worth evaluating for that project.
twittermachine-learningoptimizersweight-decayadamwnanogpttraining-techniques
Kevin Frans ✅ @kvfrans · 16h
This is a great investigation, and it answers the itch that often comes up when using gradient accumulation -- why not just make more frequent updates? The answer is that you need to properly scale the b1/b2 settings too!
> QUOTED: Micah Gold... @micahgold... · 20h
> 🚨 Did you know that small-batch vanilla SGD without momentum (i.e. the first optimizer you learn about in intro ML) is virtually as fast as AdamW for LLM pretraining on a per-FLOP basis? ...
> [Line chart titled "GPT-3 (1.3B)": FineWeb loss vs training tokens (0–10B) for two curves — AdamW BS=512 (blue) and SGD no momentum BS=1 (orange). Both curves track closely together, descending from ~4.3 to ~2.9-3.0 loss, with orange showing a few large early spikes.]
Note from Claude Sonnet 5
A machine-learning Twitter exchange about optimizer scaling laws — the surprising finding that small-batch vanilla SGD without momentum tracks AdamW's per-FLOP training efficiency for LLM pretraining, plus a note on scaling beta1/beta2 with gradient accumulation. Technical reading consistent with Nathan's interest in optimizer behavior for his own model training.
machine learningoptimizerssgdadamwllm pretrainingtwittertechnical discussion
```
davinci @basedneoleo · Jul 2 gonna try this rn wtf 💬1 ♡3 📊357 kalomaze @kalomaze · Jul 2 note: mistral is weird and has smaller weight distribution ranges compared to qwen or llama you may need to adjust by an OOM or two for the clip on those models, but the results transfer to them too, seemingly any adamw optimized Transformer actually 💬1 ♡7 📊339 davinci @basedneoleo · Jul 2 nice i hope it reproduces 💬1 ♡3 📊185 kalomaze @kalomaze · Jul 2 this trick i already used before for GRPO to stabilize and multiple people reached out to @willccbb claiming they could reproduce, so i'm bloomer overall on this trick transferring broadly ♡9 📊134 ueaj @_ueaj · Jul 2 what optimizer? grad clip or update clip?
💬1 🔁 ♡ 📊265 kalomaze ✅✓ @kalomaze · Jul 2 gradient norm clipping 💬1 🔁 ♡4 📊274 ueaj @_ueaj · Jul 2 whole gradient or per parameter? If per parameter then it's effectively sign grad, works b/c easy to fully cancel out if whole gradient then it's b/c of loss of prec, large components survive, rest zero out it'd be good to figure out exactly why it works for high/low prec ft 💬1 🔁 ♡5 📊130 biased estimator @selfattentive · Jul 2 prec? 💬1 🔁 ♡ 📊49 ueaj @_ueaj · Jul 2 precision (character limit moment) 💬 🔁 ♡1 📊44 Noah Vandal ✅ @noah_vandal · Jul 2 i wonder how well this would work with a model like mistral, which does not have a very spread out weight distribution
```
Note from Claude Sonnet 5
Continuation of the same ML training-technique thread (kalomaze's extreme gradient-clipping trick, tested across Mistral/Qwen/Llama and referencing GRPO training stabilization). Technical ML discussion, not AI-safety focused. A technical ML Twitter thread about gradient clipping strategies (whole-gradient vs per-parameter) and their interaction with numerical precision during fine-tuning. Reflects Nathan's ongoing interest in optimizer internals relevant to his own model training work (e.g. brain_graph_1).
machine-learningtraininggradient-clippinggrpooptimizationtechnicalmachine learningoptimizersgradient clippingfine-tuningtwittertechnical discussion
