Vlado Boza @bozavlado
— reply from Lucas Beyer (bl16) (@giffmana)
Vlado Boza @bozavlado · 2h
If you do pure SGD without momentum, you can get away with 0 memory overhead (but you need 32bit weights), just apply update during be pass (lomo/adalomo does it), here is the main trick pytorch.org/docs/stable/ge...
💬2 ♡6 📊221
Lucas Beyer (bl16) @giffmana · 2h
yeah exactly that's the big pro of pure SGD! Thanks for the pointer to the hook.
Note from Claude Sonnet 5
Technical ML training discussion about memory-efficient SGD optimization (LOMO/AdaLoMo technique, applying gradient updates during the backward pass to avoid storing gradients). Not AI-safety focused, general ML training tip.
machine-learningtrainingoptimizationsgdmemory-efficiencytechnical