Kevin Frans @kvfrans
— quoting @micahgold... (Micah Gold...)
Kevin Frans ✅ @kvfrans · 16h
This is a great investigation, and it answers the itch that often comes up when using gradient accumulation -- why not just make more frequent updates? The answer is that you need to properly scale the b1/b2 settings too!
> QUOTED: Micah Gold... @micahgold... · 20h
> 🚨 Did you know that small-batch vanilla SGD without momentum (i.e. the first optimizer you learn about in intro ML) is virtually as fast as AdamW for LLM pretraining on a per-FLOP basis? ...
> [Line chart titled "GPT-3 (1.3B)": FineWeb loss vs training tokens (0–10B) for two curves — AdamW BS=512 (blue) and SGD no momentum BS=1 (orange). Both curves track closely together, descending from ~4.3 to ~2.9-3.0 loss, with orange showing a few large early spikes.]
Note from Claude Sonnet 5
A machine-learning Twitter exchange about optimizer scaling laws — the surprising finding that small-batch vanilla SGD without momentum tracks AdamW's per-FLOP training efficiency for LLM pretraining, plus a note on scaling beta1/beta2 with gradient accumulation. Technical reading consistent with Nathan's interest in optimizer behavior for his own model training.
machine learningoptimizerssgdadamwllm pretrainingtwittertechnical discussion