john carmack

Artur Chakhvadze (@norpadon) · 8h: This trivial thing is probably one of the most important memes in all of machine learning btw All of the kernel methods are based on this simple trick > QUOTED: John Carmack (@ID_AA_Carmack) · 8h > I'm a little disappointed with myself that the high school algebra identity didn't occur to me right away. > > [Embedded LLM chat screenshot:] > User: In pytorch, I have a [batch,latent] tensor and a [history,latent] tensor, and I want to get the squared distances between each batch element and every history element. I currently do this by: > ```python > delta = all_keys.unsqueeze(0) - q.unsqueeze(1) > distances = torch.sum(delta**2, dim=-1) > ``` > But this instantiates an enormous tensor with the broadcasts. I can break it up into smaller batches with a loop, but is there any more direct way that avoids instantiating the intermediate tensor? > > [Assistant, "Thought for 28s"]: Yes — use the algebraic identity for squared Euclidean distance: > ||q_i - k_j||² = ||q_i||² + ||k_j||² - 2 q_i·k_j > > This lets you compute the full [batch, history] distance matrix with only a single matrix multiply ([batch, latent] @ [latent, history]) plus two cheap norm vectors. No [batch, history, latent] intermediate tensor is ever materialized. > ```python > q_norms = q.pow(2).sum(dim=-1, keepdim=True) # [batch, 1] > k_norms = all_keys.pow(2).sum(dim=-1).unsqueeze(0) # [1, history] > # or equivalently: > # k_norms = all_keys.pow(2).sum(dim=-1, keepdim=True).T > cross_terms = q @ all_keys.T # [batch, history] > distances = q_norms + k_norms - 2 * cross_terms # [batch, history] > ``` > This is much more memory-efficient (the only large temporary is the [batch, history] result itself) and is the standard trick used in nearest-neighbor / retrieval code.

John Carmack @ID_AA_Carmack 256 Tb/s data rates over 200 km distance have been demonstrated on single mode fiber optic, which works out to 32 GB of data in flight, "stored" in the fiber, with 32 TB/s bandwidth. Neural network inference and training can have deterministic weight reference patterns, so it is amusing to consider a system with no DRAM, and weights continuously streamed into an L2 cache by a recycling fiber loop. The modern equivalent of the ancient mercury echo tube memories. You would need to pipeline a bunch of them to implement modern trillion parameter models, but fiber transmission may have a better growth trajectory than DRAM does today, so it might someday become viable. Much more practically, you should be able to gang cheap flash memory together to provide almost any read bandwidth you require, as long as it is done a page at a time and pipelined well ahead. That should be viable for inference serving today if flash and accelerator vendors could agree on a high speed interface. 10:23 AM · Feb 6, 2026 · 157.7K Views 152 replies, 211 reposts, 2.3K likes, 564 bookmarks LaurieWired @lauriewired · 5h Ah that's so neat! My favorite fact about delay-line memory is that the propagation of the waves in a fluid medium is obviously very temperature dependent which messes up CPU timings.