architecture research

3 captures, most recent first.

2026-05-21

catid @MrCatid · 9h So now that we know intelligence (planning, reasoning, etc) is just finding efficient representations, who's working on a new type of ML model that learns representations? Seems like we need something new closer to VQVAE with an evolutionary rule

Note from Claude Sonnet 5

A tweet speculating about future ML architectures for representation learning (VQVAE-like with evolutionary rules). Technical AI research commentary, tangentially related to Nathan's own architecture work on brain_graph_1.

twitter machine learning representation learning vqvae architecture research

ueaj @_ueaj

2026-04-21

ueaj @_ueaj · 18h New blog! You can just keep increasing the amount of heads in your model with no diminishing returns on ICL up to atleast 4x. For reference that would make the o_proj head dimensions in this experiment 16k x 2k. Additionally, if you perform a truncated SVD on full rank master weights to train MLA instead of training them as two separate matrices, you can recover most of the ICL capability but with less memorization. I think MLA specialized optimizers are a direction worth exploring and are very underserved rn. Unfortunately I have more important projects to attend to and I've burned like 300$ on compute for this already. I would highly recommend someone trying to scale this up and see how well we can do. [Chart: "arc/output_loss" — lines for mla-rat-l512-r64-b0p9-arc1-200m, mla-base-l512-r64-b0p9-arc1-200m, quad-kv-b0p9-arc1-200m, single-kv-b0p9-arc1-200m, x-axis train/tokens_seen 20M-180M, y-axis loss 0-8, all converging to similar low loss; tooltip shows single-kv-b0p9-arc1-200m: 0.74451, quad-kv-b0p9-arc1-200m: 0.66843 at 199,884,800 tokens] ueaj @_ueaj · Apr 15 Replying to @_ueaj Also something like MLA should be trained like QAT but instead of converting a high precision matrix to low precision you convert a full rank master weights into low rank latent projections...

Note from Claude Sonnet 5

A technical thread on Multi-head Latent Attention (MLA) architecture experiments — scaling attention heads without diminishing ICL returns, and using truncated SVD on full-rank weights (analogous to quantization-aware training) to recover in-context-learning capability with less memorization. Directly relevant to the brain_graph_1 "KDA-MLA attention at the ends only" architecture plan noted in project memory.

machine learning attention mechanisms mla multi-head latent attention in-context learning twitter ueaj brain_graph_1 architecture research

Albert Gu @_albertgu

2026-04-17 — quoting Hayden Prairie (@hayden_prairie)

Albert Gu @_albertgu · Apr 16 a dynamical systems point of view, which looks like an SSM applied along the residual stream, informs more principled ways to scale looped architectures > QUOTED: > Hayden Prairie @hayden_prairie · Apr 15 > We've been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters. > > Turns out the answer is YES – by looping blocks… > > [Diagram: recurrence block h_t → Ā, Δ, B̄ (SSM-style gating) feeding into a "Transformer Blocks" module (Multi-Head Attention + Feed Forward, residual connections) → h_t+1] > [Two plots: "IsoFLOP Scaling Laws of Looping" — Validation Loss vs Recurrence (multiple colored curves with star-marked optima), and Optimal Recurrence vs FLOPs (log-log, roughly linear trend)]

Note from Claude Sonnet 5

Albert Gu (Mamba/SSM researcher) comments on a scaling-laws paper/thread about looping transformer blocks with an SSM-style gate applied along the residual stream, treating it as a dynamical-systems framing for how much recurrence to use at a given FLOP budget. Directly relevant to the brain_graph_1 encode/iterate-in-latent/decode architecture and its "virtualizing width via looping" primitive — a scaling-law reference for how many DEQ/loop iterations to use at different compute scales.

machine learning ssm mamba looped transformers scaling laws twitter albert gu architecture research brain_graph_1