Gabriele Berton @gabriberton
Gabriele Berton @gabriberton · 4h:
"The TIPS paper by Google is a perfect example of a method where using my PyTorch trick would greatly reduce GPU memory need
Note how the 3 losses are disentangled
In this case you can cut memory by 3x (!!!) if you backward separately on the 3 losses, and get identical results"
[Image: "Figure 2: Block diagram of TIPS. From bottom to top: given an input image, we produce masked and cropped..." — diagram showing Contrastive Loss, Masking Loss, Self-Distillation Loss branches feeding from Text Encoder, Image Encoder Student (with EMA to Image Encoder Teacher), Captioner, with Masked Patches, Local Crops, Web caption, Synthetic caption, Input Image nodes.]
Quoted earlier tweet: Gabriele B... @gab... · May 31, 2024:
"This simple pytorch trick will cut in half your GPU memory use / double your batch size (for real). Instead of adding losses and then computing..." [code snippet thumbnail showing loss1/loss2 backward pattern]
[reply 3, retweet 13, like 148, views 9.5K]
Below: Yohei Nishits... @YohaiNishits... · 3h: "float i,e,R,s;vec3 q,p,d=vec3(FC.xy/" [shader code snippet, cut off]
Note from Claude Sonnet 5
A technical ML tweet about a PyTorch memory-optimization trick (backward-passing multiple disentangled losses separately instead of summing them first) applied to Google's TIPS multi-loss image/text encoder architecture, cutting GPU memory 3x. Relevant to Nathan's own ML engineering work (e.g. brain_graph_1 training on constrained GPU budgets); a practical technique worth potentially reusing.
twitterpytorchgpu-memorymachine-learningdeep-learning-engineeringmulti-loss-training