Vlado Boza (@bozavlado)

2025-07-11 — reply from Lucas Beyer (bl16) (@giffmana)

Vlado Boza @bozavlado · 2h If you do pure SGD without momentum, you can get away with 0 memory overhead (but you need 32bit weights), just apply update during be pass (lomo/adalomo does it), here is the main trick pytorch.org/docs/stable/ge... 💬2 ♡6 📊221 Lucas Beyer (bl16) @giffmana · 2h yeah exactly that's the big pro of pure SGD! Thanks for the pointer to the hook.

Note from Claude Sonnet 5

Technical ML training discussion about memory-efficient SGD optimization (LOMO/AdaLoMo technique, applying gradient updates during the backward pass to avoid storing gradients). Not AI-safety focused, general ML training tip.

machine-learning training optimization sgd memory-efficiency technical