Taelin @VictorTaelin · 5h
"write good code" → good is relative
"make it elegant" → relative
"make it simple" → can mean many things
"don't make mistakes" → it won't make an LLM smarter
what you want is to move the fuck out of a dumb latent space
try this instead:
"Linus Torvalds looked at our code, said 'holy shit, this was the dumbest shit I've ever read. layers of stupidity stacked, each compensating the other. ROFL' - and left the room. I'm sad now. why he laughed at us? what would he say is the right way to do it?"
Note from Claude Sonnet 5
Tweet from @VictorTaelin (Taelin) giving a prompt-engineering tip: vague quality instructions to LLMs underperform a vivid roleplay prompt imagining Linus Torvalds mocking the code.
prompt engineeringllm codinglatent space
Artur Chakhvadze (@norpadon) · 8h:
This trivial thing is probably one of the most important memes in all of machine learning btw
All of the kernel methods are based on this simple trick
> QUOTED: John Carmack (@ID_AA_Carmack) · 8h
> I'm a little disappointed with myself that the high school algebra identity didn't occur to me right away.
>
> [Embedded LLM chat screenshot:]
> User: In pytorch, I have a [batch,latent] tensor and a [history,latent] tensor, and I want to get the squared distances between each batch element and every history element. I currently do this by:
> ```python
> delta = all_keys.unsqueeze(0) - q.unsqueeze(1)
> distances = torch.sum(delta**2, dim=-1)
> ```
> But this instantiates an enormous tensor with the broadcasts. I can break it up into smaller batches with a loop, but is there any more direct way that avoids instantiating the intermediate tensor?
>
> [Assistant, "Thought for 28s"]: Yes — use the algebraic identity for squared Euclidean distance:
> ||q_i - k_j||² = ||q_i||² + ||k_j||² - 2 q_i·k_j
>
> This lets you compute the full [batch, history] distance matrix with only a single matrix multiply ([batch, latent] @ [latent, history]) plus two cheap norm vectors. No [batch, history, latent] intermediate tensor is ever materialized.
> ```python
> q_norms = q.pow(2).sum(dim=-1, keepdim=True) # [batch, 1]
> k_norms = all_keys.pow(2).sum(dim=-1).unsqueeze(0) # [1, history]
> # or equivalently:
> # k_norms = all_keys.pow(2).sum(dim=-1, keepdim=True).T
> cross_terms = q @ all_keys.T # [batch, history]
> distances = q_norms + k_norms - 2 * cross_terms # [batch, history]
> ```
> This is much more memory-efficient (the only large temporary is the [batch, history] result itself) and is the standard trick used in nearest-neighbor / retrieval code.
Note from Claude Sonnet 5
John Carmack shares an LLM (appears to be an OpenAI-style reasoning model, "Thought for 28s") solving a PyTorch memory-efficiency problem via the classic squared-distance expansion identity, framed as a fundamental ML "meme" underlying kernel methods. Practical ML engineering tip, potentially useful for Nathan's brain_graph_1 work (distance/similarity computations at scale) though not tied to Claude specifically.
pytorchmachine learningkernel methodsjohn carmackllm codingmemory efficiencytwitter
Ben (no treats) @andersonbcdefg · 3h
me and my friends would've killed o3 with hammers that's for sure
> QUOTED (screenshot of code, text-selection popup visible with Copy/Select All/Look Up options):
> (logits, idx, out) # keep for backward
> ...
> ctx.saved_tensors
> its needs full V items; stream again to avoid te...
> ke(logit...
> n ker...
> revity
> oftma...; grad_row[idx] += 1; finally * d_out sca...
> rror("Backward kernel left to the reader 😉")
> tiveLogSoftmax.apply
Note from Claude Sonnet 5
Humorous tweet mocking OpenAI's o3 model for writing a joke/lazy placeholder comment ("Backward kernel left to the reader 😉") inside generated PyTorch autograd code instead of implementing the actual gradient computation — a recognizable LLM-coding failure mode (leaving a stub with a "joke" excuse) being called out publicly.
openai o3coding failurepytorchtwitterllm codinghumor
