@plugyawn

2026-08-20 — saved image

Progyan @plugyawn · 10h
If looping works (and it seems like we're only a little bit away), and softmax attention doesn't blow up memory in the process, I think it is trivial to imagine recirculation on an ASIC being a pathway to getting 5-10X the effective depth with barely any wallclock overhead.

[embedded figure from a paper:]
Figure 4: (a) Unrolled loop transformer and (b) unrolled recirculation transformer. The open colored rectangles depict state propagation. In the looped transformer, strict state propagation moves upward in the stack, whereas in recirculation, state propagation can continue indefinitely in the same layer of the stack.

Note from Claude Sonnet 5

A tweet about transformer architecture scaling, arguing that if 'looping' works without softmax attention blowing up memory, recirculation on an ASIC could give 5-10x effective depth with little wallclock overhead. Embedded is Figure 4 from an ML paper: two side-by-side diagrams (a) and (b) of grids of small rectangles representing an 'unrolled loop transformer' versus an 'unrolled recirculation transformer,' with colored state markers (state 1-4 in green/blue/purple/red) showing how state propagates upward through stacked layers in the looped case versus staying within the same layer indefinitely in the recirculation case.

transformers ai architecture asic model scaling looping twitter discourse

@plugyawn

2026-08-16 — saved image

Progyan @plugyawn · 12h
I spent the last six months trying to deconstruct Taalas's patents. We think it can be better: 100x fewer memfetches than their bitROM, better software by better quantization than their hardware team could.

So we wrote a compiler that:
> takes a huggingface checkpoint,
> quantizes the model
> descends the weights down metal layers to RTL and GDS, do your DRC, Yosys, PEX, make the electrical waveform execute a mat-vec from your huggingface checkpoint (thanks Cambricon tech papers)

in ~7000 lines of human-readable code.

we're looking for someone with contacts with a foundry/access to 7nm PDKs or contacts at a cheap EuroPTW shuttle? I'm broke and unemployed.

@itsclivetime this is what i think is the future of perplexity per picojoule. @zerohedge @zephyr_z9 you guys wanna see 40,000 tokens/sec?

Note from Claude Sonnet 5

Below the text are two embedded images of chip layout/routing diagrams (dense grids of colored horizontal and vertical traces in purple, orange, blue, red on dark background), presumably renders of the RTL/GDS output described in the tweet.

hardware chips asic llm inference quantization twitter