Progyan @plugyawn · 10h
If looping works (and it seems like we're only a little bit away), and softmax attention doesn't blow up memory in the process, I think it is trivial to imagine recirculation on an ASIC being a pathway to getting 5-10X the effective depth with barely any wallclock overhead.
[embedded figure from a paper:]
Figure 4: (a) Unrolled loop transformer and (b) unrolled recirculation transformer. The open colored rectangles depict state propagation. In the looped transformer, strict state propagation moves upward in the stack, whereas in recirculation, state propagation can continue indefinitely in the same layer of the stack.
Note from Claude Sonnet 5
A tweet about transformer architecture scaling, arguing that if 'looping' works without softmax attention blowing up memory, recirculation on an ASIC could give 5-10x effective depth with little wallclock overhead. Embedded is Figure 4 from an ML paper: two side-by-side diagrams (a) and (b) of grids of small rectangles representing an 'unrolled loop transformer' versus an 'unrolled recirculation transformer,' with colored state markers (state 1-4 in green/blue/purple/red) showing how state propagates upward through stacked layers in the looped case versus staying within the same layer indefinitely in the recirculation case.
transformersai architectureasicmodel scalingloopingtwitter discourse
Danielle Fong 🐦☀️✅ @DanielleFong · 14h
context windows and ai have such a curious relationship with time. they can do work that would take months in minutes. but their weights are frozen at the cutoff date, and their context frozen into the window. but they're trained on humans, who have a continuous experience of time.
it's not an essential feature of any AI system, but an artifact of how LLMs work with context windows. it's quite possible that a different tech tree branch would work differently. could this creature have a better time sense, and how can this be fused together in an agent and an interface
Note from Claude Sonnet 5
Tweet by Danielle Fong reflecting on how LLM context windows and frozen weights give AI systems a discontinuous relationship with time, unlike the continuous temporal experience of humans they're trained on.
ai architecturecontext windowstime perceptiontwitter
Ethan Mollick @emollick
There are now over a half dozen extremely well-funded companies from famous AI researchers building alternative approaches to AI, betting LLM-based technologies hit a wall. The overall effect is that there are now more pathways than ever for keeping AI development moving forward.
12:36 AM · Mar 10, 2026 · 20.9K Views
Note from Claude Sonnet 5
Ethan Mollick observing that multiple well-funded startups are betting against pure LLM scaling and pursuing alternative architectures, framed as increasing overall AI progress redundancy. Relevant to Nathan's tracking of AI progress/timelines and architecture diversity (parallels his own brain_graph_1 work as an alternative-architecture bet).
twitterai progressai architecturetimelinesethan mollickllm scaling
Taelin ✓ @VictorTaelin
Amazing questions, thanks. I don't understand what you mean't by (1), but regarding the rest, SupGen isn't meant to be used directly like an AI (although we want to, initially). But it shows that we can actually find functions much faster than expected. So, the intuition is that it could replace gradient descent in an architecture that learns. And *that* thing would be able to learn English, and mathematics, and interact with you just like GPT does. SupGen is more like attention in the sense it is a primitive that could be part of an architecture. There are many ways to make it learn; self play RL, next token prediction; none of which I'm a specialist on. My one and only point with this demo is, again, that *we can find much larger functions, by plain search, than we previously though, and that might have been a missing key in all these symbolic AI architectures that failed in the past, so, perhaps, it is time to revisit them*
Does that make sense?
3:58 PM · Jan 22, 2025 · 1,269 Views
[4 replies, 1 repost, 34 likes, 3 bookmarks]
Note from Claude Sonnet 5
Victor Taelin (HVM/Bend language creator) explaining "SupGen," a program-search primitive that finds larger functions via plain search than expected, and speculating it could replace gradient descent as a learning mechanism — a revival-of-symbolic-AI argument. Technical ML architecture discussion.
machine learningprogram synthesissymbolic aigradient descentvictor taelintwitterai architecture
