← All topics

mla

2 captures, most recent first.

@firesidealpha

— saved image

Fireside Alpha @firesidealpha · 7h
SemiAnalysis' Jordan Nanos reveals the OpenAI engineers scrolling their own kernel code had no idea what it did line by line, and it didn't matter because the AI does

"It's not slop, because it performs, but it's literally just AI-generated assembly basically puked out in Gluon, this low-level kernel programming language that they built on top of Triton, which uses this really interesting programming model."

"The point is, the guys who were scrolling through this code with us, it was kind of clear that they know a whole bunch about hardware, they know a whole bunch about the concepts in the system, and they just have no idea what this MLA kernel that they're showing us for DeepSeek actually does."

"You can go line by line and it's like nope, nope, nope."

"But it doesn't matter. The AI understands it, the AI tests it, and you see the results. It produces correct kernels that perform really well."

"And I think this is just a sign of what's to come again."

"That actual code is not necessarily something that a human has to reason about deeply, if the AI knows how to manipulate the data movement and the processing elements on the hardware that you've given it."

________ [cut off]
Note from Claude Sonnet 5

X post by Fireside Alpha quoting SemiAnalysis' Jordan Nanos on OpenAI engineers not understanding AI-generated Gluon/Triton MLA kernel code line by line, with the AI testing it and the results being correct. Capture ends at a divider line at the bottom edge.

ai-generated codekernelsgluontritonsemianalysisopenaideepseekmla

ueaj @_ueaj

ueaj @_ueaj · 18h New blog! You can just keep increasing the amount of heads in your model with no diminishing returns on ICL up to atleast 4x. For reference that would make the o_proj head dimensions in this experiment 16k x 2k. Additionally, if you perform a truncated SVD on full rank master weights to train MLA instead of training them as two separate matrices, you can recover most of the ICL capability but with less memorization. I think MLA specialized optimizers are a direction worth exploring and are very underserved rn. Unfortunately I have more important projects to attend to and I've burned like 300$ on compute for this already. I would highly recommend someone trying to scale this up and see how well we can do. [Chart: "arc/output_loss" — lines for mla-rat-l512-r64-b0p9-arc1-200m, mla-base-l512-r64-b0p9-arc1-200m, quad-kv-b0p9-arc1-200m, single-kv-b0p9-arc1-200m, x-axis train/tokens_seen 20M-180M, y-axis loss 0-8, all converging to similar low loss; tooltip shows single-kv-b0p9-arc1-200m: 0.74451, quad-kv-b0p9-arc1-200m: 0.66843 at 199,884,800 tokens] ueaj @_ueaj · Apr 15 Replying to @_ueaj Also something like MLA should be trained like QAT but instead of converting a high precision matrix to low precision you convert a full rank master weights into low rank latent projections...
Note from Claude Sonnet 5

A technical thread on Multi-head Latent Attention (MLA) architecture experiments — scaling attention heads without diminishing ICL returns, and using truncated SVD on full-rank weights (analogous to quantization-aware training) to recover in-context-learning capability with less memorization. Directly relevant to the brain_graph_1 "KDA-MLA attention at the ends only" architecture plan noted in project memory.

machine learningattention mechanismsmlamulti-head latent attentionin-context learningtwitterueajbrain_graph_1architecture research