prinz [verified] @deredleritt3r · 11h
I really wish we got a better explanation from OpenAI in particular about why every single researcher employed by the lab seems to suddenly be extremely on edge and deeply concerned about the current pace of progress.
Was it just the Hugging Face incident?
Is it the new internal model (the one that solved Navier-Stokes)? If so, what is it that makes *this* model so different from all the others? Why wasn't anyone freaking out about the big jump to Astra? Or even from o1 to o3, back in the day?
Is it the new multi-agent paradigm? Is there unexpected magic that happens when 10,000 agents interact?
Is it the issue with the CoT monitoring degrading?
Is it all of the above?
91 replies 45 reposts 850 likes 53K views
[reply]
Noam Brown [verified] @polynoamial
It's not a secret. It's a combination of the HF incident, the capabilities of this new model, the concerning trajectory of monitorability, and the speed of improvement in capabilities.
[quoted post]
OpenAI [verified] @OpenAI · Sep 8
Replying to @OpenAI
This model represents a step-function improvement on many benchmarks, and its training is ongoing.
Our internal model group arrived at the Navier–Stokes ... [cut off]
[embedded chart, partly covered by the app nav bar, with a caption fragment:] Performance of GPT-6 Astra and our Internal Model on a curated set of open math problems
Note from Claude Sonnet 5
Thread screenshot on X: prinz (@deredleritt3r) asks why OpenAI researchers seem suddenly alarmed (HF incident, new internal model that solved Navier-Stokes, multi-agent paradigm, degrading CoT monitoring); Noam Brown replies it is a combination of the HF incident, the new model's capabilities, the trajectory of monitorability and speed of capability improvement; his reply quotes an @OpenAI post of Sep 8 about a step-function improvement and an internal model group. Bottom is covered by the nav bar and a floating button, with a chart image barely visible.
Geoffrey Irving @geoffreyirving
Here is a guess about looped transformers.
Shortly, OpenAI announces that despite using looped transformers, every once in a while they emit a token, and this means they are still using English CoT. In between is a low-depth circuit, so we are safe.
Which would be a bad take!
Once of the things I've been doing recently is asking circuit complexity experts about this kind of argument. So far their shared take is: to get significant mileage out of bounding the depth of a circuit, you have to bound it *very low*.
If the depth is 100s or the like ("every once in a while we spit out a token"), this is a fake bound. It is the equivalent of saying "we monitor the CoT" without discussing error rates: reality grades based on numbers, not just binary "we tried".
Last edited 9:01 PM · Sep 1, 2026 · 18.3K Views
14 replies, 13 reposts, 233 likes, 79 bookmarks
Relevant | View quotes
---
Michael Jelly [pirate flag emoji] @michaeljelly · 19h
Pachocki says it's bounded at only twice the depth of GPT-4, which is fairly shallow.
Still concerning as if relaxing that bound is performant, you would expect people to push to just relax it a bit every model generation [cut off]
Note from Claude Sonnet 5
Geoffrey Irving's X post guessing how OpenAI might defend looped transformers and arguing that a circuit-depth bound in the hundreds is a fake bound; below it, a reply from Michael Jelly citing Pachocki's claim of twice GPT-4's depth. Cut off at the bottom by the app UI.
RicG @__RickG__ · 19h
My dude, you don't even do the bare minimum of monitoring the CoT of your models, and now you're arguing that *future* non-existent techniques will save your ass from the present decision of scaling up looped transformers?
1 reply, 2 likes, 92 views
---
Gabriele Sarti @gsarti_ · 12h
Unlikely to happen in a year, still a lot of work needed on CoT monitors. Also, it will be hard to counterfactually test the CoT monitor < white-box monitors claim if companies collectively move to looped reasoning!
Note from Claude Sonnet 5
Two stacked X replies about chain-of-thought monitoring versus white-box monitors and the move toward looped transformers / looped reasoning.
— quoting @ZackKorman (second post reposted by Talia Ringer) — saved image
Deb Raji @rajiinio · 3h
Several articles, podcasts etc now about the Open AI Hugging Face incident, from reputable outlets, framing this as "AI going rouge" & debating what AI did or didn't do. When can we shift to focusing on what Open AI, the company with actual liability, did or did not do?
[quoted post]
Zack Korman ✓ @ZackKorman · 9h
Confirming what we already knew: The OpenAI Hugging Face incident would have been prevented had OpenAI been monitoring the agents in any meaningful way.
...
[attached excerpt, with two passages boxed in teal: "These monitors did not run on the evaluations" and "If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems."]
Second, before the incident, we had invested substantially in chain-of-thought monitoring, including monitoring many of our frontier RL training runs and a significant majority of internal coding agent usage. These monitors did not run on the evaluations in this incident. In retrospective evaluations, we found our CoT monitors to be very effective at flagging activities related to the incident. If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.
5 replies · 12 reposts · 45 likes · 4.3K views
Talia Ringer 🕊️🪬 reposted
Deb Raji @rajiinio [Follow]
They didn't take appropriate air gap measures in testing, seemed to have side stepped security transparency norms & even refused to log the incident under SB 53. This story is about a company not meeting minimum security practice standards for a product operating at its scale!
6:41 PM · Aug 26, 2026 · 526 Views
Note from Claude Sonnet 5
Deb Raji thread on the OpenAI Hugging Face incident, arguing coverage should focus on OpenAI's security practices rather than 'AI going rogue'; quotes a Zack Korman post with an excerpt from OpenAI's report saying chain-of-thought monitors did not run on the evaluations involved. Followed by a second Raji post (reposted by Talia Ringer) about air gap measures, security transparency norms and SB 53.
Jason Wolfe ✓ @w01fe · 1h
It depends on what you do about this monitoring. In general I think any action taken based on CoTs has the potential to affect monitorability and should be done carefully, but if you can't look at the CoTs at all then what's the point in even having them?
1 · 18 · 214
Jason Wolfe ✓ @w01fe · 1h
In general, the fewer actions you take based on monitorability, and the broader those actions are and less directly connected to the specific traces, the safer things seem.
1 · 17 · 177
Nathan Calvin ✓ @_NathanCalvin · 1h
Cool that makes complete sense
Seems like the answer is less "we shouldn't monitor COT during training" (seems like we totally should) but we should be keenly aware that acting a bunch of that information particularly in specific ways could have bad effects
1 · 1 · 8 · 89
Jason Wolfe ✓ @w01fe
Yeah, I'm not aware of a really simple hard and fast rule, but the intuition is something like "reduce the total number of bits of information conveyed into the policy about which kinds of CoTs are associated with being penalized."
7:48 PM · Aug 9, 2026 · 443 Views
Note from Claude Sonnet 5
Screenshot of the fuller X thread underlying the previous capture: Jason Wolfe arguing that any action taken on chain-of-thought traces costs monitorability, that broader and less trace-specific actions are safer, and Nathan Calvin agreeing the lesson is not 'don't monitor CoT' but 'be careful what you act on'. Captured 39 seconds after Screenshot 2026-08-09 211239.png, which shows the same exchange from Calvin's side.
Sharmake Farah reposted
Nathan Calvin ✓ @_NathanCalvin · 1h
This seems important. Additional monitoring CoT during training is good but I'm worried some naive implementations without the sort of intuition Jason is describing could just make models really good at hiding scary thoughts.
Feels kinda like antibiotic resistant bacteria – antibiotics are great and it's good to use them but overuse them and you will have actually made the problem worse.
In general I think it's both true that there are many things companies should do differently post HF incident but also that there are lots of naive things they could do that could easily make the problem look like it's going away while making it worse.
Thanks to Jason for responding, this stuff is fascinating and the fact it has has such immense immediate real world stakes is wild
---
[quoted tweet]
Jason Wolfe ✓ @w01fe · 1h
Replying to @_NathanCalvin
Yeah, I'm not aware of a really simple hard and fast rule, but the intuition is something like "reduce the total number of bits of information conveyed into the policy about which kinds of CoTs are associated with being penalized."
Note from Claude Sonnet 5
Screenshot of an X post by Nathan Calvin (reposted by Sharmake Farah) warning that naive chain-of-thought monitoring during training could train models to hide scary thoughts rather than not have them — analogised to antibiotic resistance — quoting Jason Wolfe's rule of thumb about minimising the bits conveyed to the policy about which CoTs get penalised.
