@dnhkng

**David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681648096788749) 4/n Clue #2: Alpindale's Goliath-120B. A frankenmerge that fed the OUTPUT of layer 16 of one model into the INPUT of layer 8 of another. Distributions the model never saw during training. It should have been gibberish but it worked. Layers are more interchangeable than expected. --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681650080723245) 5/n So I had a hypothesis: Transformers have a genuine functional anatomy. Early layers encode. Late layers decode. Middle layers reason in a universal internal space. Maybe I didn't need to teach a model new facts. I just needed to give it more time to think. --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681651846451663) 6/n I built a "brain scanner." For each pair (i,j), duplicate layers i through j and run the whole model. For Qwen2-72B (80 layers) that's 3,241 configurations. Full leaderboard benchmarks would take a decade on my rig. I needed fast proxy tasks. --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681653700399605) 7/n After months of dead ends (creativity scoring, LLM-as-judge pipelines), I settled on two probes: Hard math: "What is the cube root of 74,088,893,247?" No chain-of-thought. Just guess. EQ-Bench: predict emotional intensity in social scenarios. Maximally orthogonal. Tiny output --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681655587823879) 8/n Days of continuous GPU time on two RTX 4090s in my basement. The optimal config: duplicate layers 45–51. Seven extra layers near the middle of the stack. 72B → 78B params, all exact copies. Zero new weights. R-epeat Y-our S-elf → RYS-XLarge. --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681658548994310) 9/nSubmitted to the leaderboard. Waited a month (the queue was brutal). Results: +17.7% on MuSR, +8.2% on MATH, 5/6 benchmarks improved. Average: #1. I never optimized for ANY leaderboard benchmark. It was pure out-of-sample validation. [image] --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681662240010407) 10/n But the heatmaps told an even better story. They're basically functional MRIs of a Transformer while it's thinking. Each pixel = one full evaluation of a re-layered model. [image] --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681665352151369) 11/n Key finding: duplicating a SINGLE middle layer almost never helps. Usually makes things worse. But duplicating a BLOCK of ~7 layers? Big boost. The middle layers aren't doing independent iterative refinement. They're circuits — multi-step recipes that work best as units. [image] --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681669265367142) 12/n The circuits have sharp boundaries. Too few layers: you've cut into the circuit. Too many: you've included tissue from a neighboring circuit. Pre-training carved these structures. They only work whole - you CAN run the whole recipe twice. [image] --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681672180482191) 13/n The bad configurations went properly unhinged. One model cheerfully announced "Let's act like cowboys! Yeehaw!" and descended into pages of hahaha interspersed with cowboy references. Not "slightly worse" Brain damage. Specific neurological deficits from disrupted circuits --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681674063749208) 14/n The method is orthogonal to fine-tuning. Layer duplication changes architecture; fine-tuning changes weights. You can stack them. As of early 2026, the top 4 models on the Open LLM Leaderboard are ALL descendants of RYS-XLarge, fine-tuned on top of the duplicated layers. --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681676232216776) 15/n The duplicated layers use no extra VRAM — they're pointer copies. More compute and KV cache, yes, but no extra memory. And I suspect fine-tuning just the two junction layers (where the loop reconnects) is all you really need to clean it up. --- **David** @dnhkng [2026-03-22](https://x.com/dnhkng/status/2035681678266347892) 16/n 20 years ago I was a PhD student dissecting rat brains. I never expected to end up performing brain surgery on artificial minds. Full writeup with all the heatmaps, math, and code details: 🔗 --- **Diego** @didacum333 [2026-03-22](https://x.com/didacum333/status/2035768422772899893) This IS incredibly interesting, so Transformers automatically create modules for reasoning and computation from scratch? What would happen if you add another module? --- **s3nh** @s3nhxx [2026-03-22](https://x.com/s3nhxx/status/2035775142131908731) @mulamx this is had to link you --- **Cynical Optimist** @ChemPhysMajor [2026-03-22](https://x.com/ChemPhysMajor/status/2035748042981163164) This is super interesting and I sincerely hope you tackle some newer models with a similar approach. I imagine this is best done with dense, MoEs presumably would need to be approached per-expert. It opens some doors into fascinating questions: how many encode and decode layers --- **poshlain** @poshlain [2026-03-22](https://x.com/poshlain/status/2035746657774542890) ( o.o ) neat! ... what do you think you should look into next?? this seems like a rly cool research angle, i'm sure you got a bunch of ideas from your progress alredy!..?? --- **Johnny Yukari** @JYukariHero [2026-03-22](https://x.com/JYukariHero/status/2035718617292046713) No training. No gradients. Just architectural surgery. Fine-tuning industry might want to take notes.