pytorch

● Major finding already: the log shows "VRAM after setup: 1411MB" — so params + optimizer + buffers are only ~1.4 GB. The OOM at 22.96 GB is almost entirely train-tick activations / the three_block —autograd graph (~21 GB). The static base is not the problem; the activation memory is. (My helper

Artur Chakhvadze (@norpadon) · 8h: This trivial thing is probably one of the most important memes in all of machine learning btw All of the kernel methods are based on this simple trick > QUOTED: John Carmack (@ID_AA_Carmack) · 8h > I'm a little disappointed with myself that the high school algebra identity didn't occur to me right away. > > [Embedded LLM chat screenshot:] > User: In pytorch, I have a [batch,latent] tensor and a [history,latent] tensor, and I want to get the squared distances between each batch element and every history element. I currently do this by: > ```python > delta = all_keys.unsqueeze(0) - q.unsqueeze(1) > distances = torch.sum(delta**2, dim=-1) > ``` > But this instantiates an enormous tensor with the broadcasts. I can break it up into smaller batches with a loop, but is there any more direct way that avoids instantiating the intermediate tensor? > > [Assistant, "Thought for 28s"]: Yes — use the algebraic identity for squared Euclidean distance: > ||q_i - k_j||² = ||q_i||² + ||k_j||² - 2 q_i·k_j > > This lets you compute the full [batch, history] distance matrix with only a single matrix multiply ([batch, latent] @ [latent, history]) plus two cheap norm vectors. No [batch, history, latent] intermediate tensor is ever materialized. > ```python > q_norms = q.pow(2).sum(dim=-1, keepdim=True) # [batch, 1] > k_norms = all_keys.pow(2).sum(dim=-1).unsqueeze(0) # [1, history] > # or equivalently: > # k_norms = all_keys.pow(2).sum(dim=-1, keepdim=True).T > cross_terms = q @ all_keys.T # [batch, history] > distances = q_norms + k_norms - 2 * cross_terms # [batch, history] > ``` > This is much more memory-efficient (the only large temporary is the [batch, history] result itself) and is the standard trick used in nearest-neighbor / retrieval code.

Gabriele Berton @gabriberton · 4h: "The TIPS paper by Google is a perfect example of a method where using my PyTorch trick would greatly reduce GPU memory need Note how the 3 losses are disentangled In this case you can cut memory by 3x (!!!) if you backward separately on the 3 losses, and get identical results" [Image: "Figure 2: Block diagram of TIPS. From bottom to top: given an input image, we produce masked and cropped..." — diagram showing Contrastive Loss, Masking Loss, Self-Distillation Loss branches feeding from Text Encoder, Image Encoder Student (with EMA to Image Encoder Teacher), Captioner, with Masked Patches, Local Crops, Web caption, Synthetic caption, Input Image nodes.] Quoted earlier tweet: Gabriele B... @gab... · May 31, 2024: "This simple pytorch trick will cut in half your GPU memory use / double your batch size (for real). Instead of adding losses and then computing..." [code snippet thumbnail showing loss1/loss2 backward pattern] [reply 3, retweet 13, like 148, views 9.5K] Below: Yohei Nishits... @YohaiNishits... · 3h: "float i,e,R,s;vec3 q,p,d=vec3(FC.xy/" [shader code snippet, cut off]

Ben (no treats) @andersonbcdefg · 3h me and my friends would've killed o3 with hammers that's for sure > QUOTED (screenshot of code, text-selection popup visible with Copy/Select All/Look Up options): > (logits, idx, out) # keep for backward > ... > ctx.saved_tensors > its needs full V items; stream again to avoid te... > ke(logit... > n ker... > revity > oftma...; grad_row[idx] += 1; finally * d_out sca... > rror("Backward kernel left to the reader 😉") > tiveLogSoftmax.apply