adamw

3 captures, most recent first.

2026-03-30

Harsh Bhatt ✓ @harshbhatt7585 · 23h I tried moun over AdamW with GRPO, it is far stable than that with moun. [2 replies, 371 views] rajan agarwal ✓ @_rajanagarwal · 23h my intuition here is that muon probably provides strong benefits when the intention is generalized RL, but there's a chance that only works with muon bases just by the argument of loss landscapes [1 reply, 321 views] Harsh Bhatt ✓ @harshbhatt7585 · 22h I tried in high dense grid world with HRM model, In my GRPO experiments, Muon felt noticeably more stable than AdamW. I also think that operator-aware step geometry may help more when the RL objective is broad and general rather than narrowly optimized, But grid-world was very narrowed as cannot be as general as LLMs but still it felt better, let me search the wandb experiment, will drop here if I find. [1 reply, 2 likes, 95 views] Harsh Bhatt ✓ @harshbhatt7585 · 22h *HRM model wa not pretrained LLM [96 views] Mingta Kaivo 明塔 ... ✓ @MingtaKa... · 12h super interesting thread. we've been experimenting with different optimizers for fine-tuning audio models at audiowave and the step size sensitivity is real. curious if anyone has tried muon specifically for RLHF on smaller models (sub-7B) or if the benefits mostly show up at scale?

Note from Claude Sonnet 5

An ML researcher discussion thread comparing the Muon optimizer vs AdamW for GRPO (a policy-gradient RL algorithm used in RLHF-style training), including stability observations across grid-world and audio model fine-tuning experiments. Technical ML-training content.

machine learning optimizers muon adamw grpo rlhf twitter reinforcement learning

Cameron R. Wolfe @cwolferesearch

2026-01-15 — quoting @varunneal

Cameron R. Wolf... ✓ @cwolfere... · Jan 14 Very rare to see useful modifications to optimizers these days. AdamW has dominated for so long with LLMs. Cautious weight decay (CWD) is a pretty simple (one-line) change to decoupled weight decay that consistently improves performance. From the paper: "Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer update." [Quoted tweet:] varun ✓ @varunneal · Jan 14 Cautious Weight Decay is a surprisingly simple technique that has been repeatedly validated in Modded NanoGPT. I expect it will gain serious traction as the default variant of decoupled weight decay x.com/Tim38463182/st... [Embedded code snippet:] apply_update(param, update, learning_rate, weight_decay): mask = (update * param) >= 0 update += weight_decay * param * mask return param - learning_rate * update

Note from Claude Sonnet 5

A technical ML tweet describing "Cautious Weight Decay" (CWD), a simple one-line modification to AdamW-style decoupled weight decay that only applies decay where the sign matches the optimizer update, validated in the Modded NanoGPT speedrun community. Relevant to Nathan's own ML/optimizer engineering work on brain_graph_1 (which is already tracking Muon and other optimizer choices per project memory) — a candidate technique worth evaluating for that project.

twitter machine-learning optimizers weight-decay adamw nanogpt training-techniques

Kevin Frans @kvfrans

2025-07-11 — quoting @micahgold... (Micah Gold...)

Kevin Frans ✅ @kvfrans · 16h This is a great investigation, and it answers the itch that often comes up when using gradient accumulation -- why not just make more frequent updates? The answer is that you need to properly scale the b1/b2 settings too! > QUOTED: Micah Gold... @micahgold... · 20h > 🚨 Did you know that small-batch vanilla SGD without momentum (i.e. the first optimizer you learn about in intro ML) is virtually as fast as AdamW for LLM pretraining on a per-FLOP basis? ... > [Line chart titled "GPT-3 (1.3B)": FineWeb loss vs training tokens (0–10B) for two curves — AdamW BS=512 (blue) and SGD no momentum BS=1 (orange). Both curves track closely together, descending from ~4.3 to ~2.9-3.0 loss, with orange showing a few large early spikes.]

Note from Claude Sonnet 5

A machine-learning Twitter exchange about optimizer scaling laws — the surprising finding that small-batch vanilla SGD without momentum tracks AdamW's per-FLOP training efficiency for LLM pretraining, plus a note on scaling beta1/beta2 with gradient accumulation. Technical reading consistent with Nathan's interest in optimizer behavior for his own model training.

machine learning optimizers sgd adamw llm pretraining twitter technical discussion