sgd

Dimitris Papailiopo... (@DimitrisPa... ✓) — 1h btw this is a weird heavy ball SGD variant that basically does this 1. Load previous checkpoint weights. 2. Reset optimizer state / momentum buffers. 3. Train N steps. 4. For first M steps: warm LR from 0.1x -> 1.0x. 5. Hold LR flat until ~50% of the cycle. 6. Linearly decay LR to zero. 7. Save checkpoint. 8. Repeat the optimizer is exactly this buf = mu * buf + grad p *= 1 - lr * wd p -= lr * buf / 524288 @CevherLIONS @_arohan_ does this have a name? I'd call it wave SGD lol

Kevin Frans ✅ @kvfrans · 16h This is a great investigation, and it answers the itch that often comes up when using gradient accumulation -- why not just make more frequent updates? The answer is that you need to properly scale the b1/b2 settings too! > QUOTED: Micah Gold... @micahgold... · 20h > 🚨 Did you know that small-batch vanilla SGD without momentum (i.e. the first optimizer you learn about in intro ML) is virtually as fast as AdamW for LLM pretraining on a per-FLOP basis? ... > [Line chart titled "GPT-3 (1.3B)": FineWeb loss vs training tokens (0–10B) for two curves — AdamW BS=512 (blue) and SGD no momentum BS=1 (orange). Both curves track closely together, descending from ~4.3 to ~2.9-3.0 loss, with orange showing a few large early spikes.]

Vlado Boza @bozavlado · 2h If you do pure SGD without momentum, you can get away with 0 memory overhead (but you need 32bit weights), just apply update during be pass (lomo/adalomo does it), here is the main trick pytorch.org/docs/stable/ge... 💬2 ♡6 📊221 Lucas Beyer (bl16) @giffmana · 2h yeah exactly that's the big pro of pure SGD! Thanks for the pointer to the hook.