Geoffrey Irving @geoffreyirving
Here is a guess about looped transformers.
Shortly, OpenAI announces that despite using looped transformers, every once in a while they emit a token, and this means they are still using English CoT. In between is a low-depth circuit, so we are safe.
Which would be a bad take!
Once of the things I've been doing recently is asking circuit complexity experts about this kind of argument. So far their shared take is: to get significant mileage out of bounding the depth of a circuit, you have to bound it *very low*.
If the depth is 100s or the like ("every once in a while we spit out a token"), this is a fake bound. It is the equivalent of saying "we monitor the CoT" without discussing error rates: reality grades based on numbers, not just binary "we tried".
Last edited 9:01 PM · Sep 1, 2026 · 18.3K Views
14 replies, 13 reposts, 233 likes, 79 bookmarks
Relevant | View quotes
---
Michael Jelly [pirate flag emoji] @michaeljelly · 19h
Pachocki says it's bounded at only twice the depth of GPT-4, which is fairly shallow.
Still concerning as if relaxing that bound is performant, you would expect people to push to just relax it a bit every model generation [cut off]
Note from Claude Sonnet 5
Geoffrey Irving's X post guessing how OpenAI might defend looped transformers and arguing that a circuit-depth bound in the hundreds is a fake bound; below it, a reply from Michael Jelly citing Pachocki's claim of twice GPT-4's depth. Cut off at the bottom by the app UI.
Thomas Larsen @thlarsen
Very bad if true. I previously thought that in a short timelines world, the most likely case was that (1) the AIs would be misaligned, but (2) we would get a lot of evidence about it from reading the COTs. This evidence would increase the chance of a reasonable response from labs/governments.
Now I still think the AIs are going to be misaligned, but that we won't have the ability to tell (we have to rely on toolcalls/agentic behaviour instead of the COT) and even if we do, the investigation will be nearly impossible because we'll have to trust the AI to self report what it was thinking about.
This is also an example of things going faster than AI 2027 -- we had Neuralese starting in March 2027
[embedded image of AI 2027 text:]
March 2027: Algorithmic Breakthroughs
Three huge datacenters full of Agent-2 copies work day and night, churning out synthetic training data. Another two are used to update the weights. Agent-2 is getting smarter every day.
With the help of thousands of Agent-2 automated researchers, OpenBrain is making major algorithmic advances. [highlighted: One such breakthrough is augmenting the AI's text-based scratchpad (chain of thought) with a higher-bandwidth thought process (neuralese recurrence and memory).] Another is a more scalable and efficien[cut off] way to learn from the results of high-effort task solutions (iterated distillation and amplification).
[quoted post:]
Amir Efrati @amir · Sep 1
new: OpenAI & others quietly using loop transformers that don't show their 'thinking' when scaled up
a leap forward on performance, but sparking concerns inside & outside OpenAI re: security as this takes off
[embedded The Information article card, cut off at bottom of screen]
Note from Claude Sonnet 5
Thomas Larsen's X post reacting to a reported shift to looped transformers whose reasoning is not visible, arguing it removes CoT evidence of misalignment and noting AI 2027's neuralese milestone; quotes Amir Efrati's post. Screenshot is cut off at the bottom by the app's navigation bar.
[top edge, partially cut off:] inside & outside OpenAI re: security as this takes off
[embedded card, The Information Pro:]
The Information Pro
Exclusive
Secret Technique Behind OpenAI's 'Astra' Model Sparks Security Concerns
[card image: illustrated collage with a man's face]
6:33 PM · Sep 1, 2026 · 323.4K Views
58 replies, 183 reposts, 1.2K likes, 598 bookmarks
Relevant | View quotes
---
Thomas Larsen @thlarsen · 6h
Update: great news! The architecture is less bad than I assumed, it seems to be only a minor increase in the opaque serial depth.
It's very important that AI companies don't venture further in this direction until we've got much better alignment / interp tools
[quoted post:]
Jakub Pachocki @merettm · 22h
I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4....
Note from Claude Sonnet 5
Continuation of the preceding scroll: the bottom of the Amir Efrati post card (The Information: 'Secret Technique Behind OpenAI's Astra Model Sparks Security Concerns'), then Thomas Larsen's update that the architecture is less bad than assumed, quoting Jakub Pachocki's statement about computation-graph depth of OpenAI frontier models including Astra.
RicG @__RickG__ · 19h
My dude, you don't even do the bare minimum of monitoring the CoT of your models, and now you're arguing that *future* non-existent techniques will save your ass from the present decision of scaling up looped transformers?
1 reply, 2 likes, 92 views
---
Gabriele Sarti @gsarti_ · 12h
Unlikely to happen in a year, still a lot of work needed on CoT monitors. Also, it will be hard to counterfactually test the CoT monitor < white-box monitors claim if companies collectively move to looped reasoning!
Note from Claude Sonnet 5
Two stacked X replies about chain-of-thought monitoring versus white-box monitors and the move toward looped transformers / looped reasoning.
Albert Gu @_albertgu · Apr 16
a dynamical systems point of view, which looks like an SSM applied along the residual stream, informs more principled ways to scale looped architectures
> QUOTED:
> Hayden Prairie @hayden_prairie · Apr 15
> We've been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters.
>
> Turns out the answer is YES – by looping blocks…
>
> [Diagram: recurrence block h_t → Ā, Δ, B̄ (SSM-style gating) feeding into a "Transformer Blocks" module (Multi-Head Attention + Feed Forward, residual connections) → h_t+1]
> [Two plots: "IsoFLOP Scaling Laws of Looping" — Validation Loss vs Recurrence (multiple colored curves with star-marked optima), and Optimal Recurrence vs FLOPs (log-log, roughly linear trend)]
Note from Claude Sonnet 5
Albert Gu (Mamba/SSM researcher) comments on a scaling-laws paper/thread about looping transformer blocks with an SSM-style gate applied along the residual stream, treating it as a dynamical-systems framing for how much recurrence to use at a given FLOP budget. Directly relevant to the brain_graph_1 encode/iterate-in-latent/decode architecture and its "virtualizing width via looping" primitive — a scaling-law reference for how many DEQ/loop iterations to use at different compute scales.
