Cameron R. Wolfe @cwolferesearch
— quoting @varunneal
Cameron R. Wolf... ✓ @cwolfere... · Jan 14
Very rare to see useful modifications to optimizers these days. AdamW has dominated for so long with LLMs. Cautious weight decay (CWD) is a pretty simple (one-line) change to decoupled weight decay that consistently improves performance.
From the paper: "Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer update."
[Quoted tweet:]
varun ✓ @varunneal · Jan 14
Cautious Weight Decay is a surprisingly simple technique that has been repeatedly validated in Modded NanoGPT. I expect it will gain serious traction as the default variant of decoupled weight decay
x.com/Tim38463182/st...
[Embedded code snippet:]
apply_update(param, update, learning_rate, weight_decay):
mask = (update * param) >= 0
update += weight_decay * param * mask
return param - learning_rate * update
Note from Claude Sonnet 5
A technical ML tweet describing "Cautious Weight Decay" (CWD), a simple one-line modification to AdamW-style decoupled weight decay that only applies decay where the sign matches the optimizer update, validated in the Modded NanoGPT speedrun community. Relevant to Nathan's own ML/optimizer engineering work on brain_graph_1 (which is already tracking Muon and other optimizer choices per project memory) — a candidate technique worth evaluating for that project.
twittermachine-learningoptimizersweight-decayadamwnanogpttraining-techniques