Minh Nhat Nguyen @menhguin
— web clipping, 226 words
Post by @menhguin on X
genuine LLM architecture question:
why does there have to be multiple sequential MLP layers? why not have many experts all communicating and routing all-to-all?
the human brain manages this fine, is multiple layers just a hack to simplify combinatorial complexity for backprop?
---
@nabla\_theta given the existing disadvantages of sparse circuits, would this possibly be less disadvantageous in a "layerless transformer"
[openai.com Understanding neural networks through sparse circuits](https://t.co/8XZelQEgqo)
---
hmmmn maybe we dont need strict layers (this neuron is always at step N in the forward pass), we might just need some kind of constraint that greatly reduces the search space to 1/n while letting backprop work, no ? theoretically if there was a search algorithm that did this without hardcoding that a neuron always has to be a certain step (which is really what layers are), it would serve just as well
sparse training costs 10x due to the gradient flow problem already, so if we're gonna have to solve the gradient flow search problem anyway, we might as well unlock layerless while we're at it. wait!
and if layer depth can be learned, we can actually reduce compute required! the attribution problem goes from a resource disadvantage (we have to search 10 layers every time) to a resource advantage (if we search, we can sometimes run backprop on fewer than 10 layers)
comparative advantage!