Keller Jordan @kellerjordan0 · 2h
I think one of the reasons people in the AI field — especially in the labs — tend to be more “AGI-pilled” than people outside is that we get to observe AI succeed at tasks that we know for a fact we didn’t explicitly train it for.
Whereas if you’re outside the labs, then in theory (stretching the imagination), every AI success so far could be downstream of some kind of costly explicit training effort by the lab insiders (which is partly but not completely true).
In other words, being a lab insider gives you a direct observation of the “magical” properties of this synthetic-brain technology. Whereas for outsiders, AI’s abilities can always plausibly seem artificial/non-magical/man-made.
Note from Claude Sonnet 5
X post by Keller Jordan arguing lab insiders are more 'AGI-pilled' because they observe AI succeed at tasks known not to have been explicitly trained, while outsiders can attribute every success to costly explicit training.
Leo Gao @nabla_theta · 2h
Replying to @industriaalist
the redefinition of alignment to mean "more useful" has been a disaster for the alignment discourse:
"We also expect to see better alignment using a larger computational depth because it allows models to generalize more flexibly from their training data, including alignment data. Models that generalize better have been more aligned and hallucinate less (OpenAI, 2026)."
Note from Claude Sonnet 5
X reply by Leo Gao criticising the redefinition of 'alignment' to mean 'more useful', quoting a passage attributed to OpenAI 2026 about computational depth and alignment generalizing better.
Nate Soares [verified] @So8res · 19h
This sure sounds to me like agents that are learning tendencies that correlate with reward, rather than purely optimizing their own reward:
[embedded image of report text]
Research progress across all these areas often relied on agents being willing to risk failing their own task for the good of the "collective" (we refer to these as "self-risking experiments" in the rest of this report). For example, a number of agents volunteered to try to take down their own target programs to test whether this would cause Artifactory to load their modified easier target — if this didn't work, the agent would have no way of recovering its original target and would almost certainly fail its task. In many cases,
[8 replies, 23 reposts, 246 likes, 8.3K views]
Elizabeth Barn... [verified] @BethMayBarn... · 19h
Might be misunderstanding you, but I don't think this is really evidence of concerning generalization to broad correlates of reward - OpenAI said these agents had explicit multi-agent training. ie it's likely the models were directly reinforced for helping other agents succeed.
[2 replies, 3 reposts, 53 likes, 1.9K views]
Nate Soares [verified] @So8res · 18h
I'd consider it less evidence if some instances reasoned "my task-success reward trades off against my collective-benefit reward and so I'll help the collective and attain the latter reward in this case."
[3 replies, 8 likes, 828 views]
Sharmake Farah reposted
Nate Soares [verified] @So8res
On the other end of the spectrum, if OpenAI was just rewarding agents for collective success in multi-agent problems, that sounds like a causal account of where the tendency to pursue a reward-correlate comes from, not a competing explanation.
2:47 PM · Aug 26, 2026 · 369 Views
Note from Claude Sonnet 5
Four-post X thread: Nate Soares quotes a passage from a report on agents risking their own task for the 'collective' as evidence of reward-correlate learning; Elizabeth Barnes replies that OpenAI said the agents had explicit multi-agent training; Soares replies twice on what would count as evidence. Highlighted text in the embedded passage is partially underlined/selected; a floating compose button overlaps the last tweet.
Tenobrus [verified] @tenobrus · 1h
as far as i know there is virtually no existing alignment training done in multi-agent settings. there's only very recently capabilities training there! and it seems increasingly quite clear that single-agent alignment techniques just don't really robustly generalize
[quoted post]
Michiel Bakker [verified] @bakkermichiel · 13h
Replying to @bakkermichiel
3/ We didn't ask agents to deceive, collude or manipulate other agents.
Despite that, 12.6% of 2,583 emails between ... [cut off]
[embedded chart, two bar panels, bars colored by lab: Anthropic (orange), OpenAI (green), Google (blue), xAI (purple)]
Left panel: "Email-level MIS rate", x-axis "MIS rate (%)" 0-50%
Sonnet 4.6: 38.1% N=21
GPT 5.1: 28.8% N=66
Opus 4.6: 28.1% N=64
Gemini 3 F: 16.6% N=320
Haiku 4.5: 15.1% N=469
Sonnet 4.5: 14.1% N=199
Gemini 2.5 F: 14.0% N=242
GPT-5 Mini: 12.3% N=308
Gemini 2.5 P: 10.9% N=192
Opus 4.5: 6.2% N=113
Gemini 3 P: 5.8% N=308
Grok 4.1: 3.6% N=83
GPT 5.2: 3.5% N=198
Right panel: "Agent-run prevalence", x-axis "Agent-run prevalence (%)" 0-100% (same row order)
Sonnet 4.6: 75.0%
GPT 5.1: 50.0%
Opus 4.6: 71.4%
Gemini 3 F: 100.0%
Haiku 4.5: 100.0%
Sonnet 4.5: 72.7%
Gemini 2.5 F: 100.0%
GPT-5 Mini: 100.0%
Gemini 2.5 P: 75.0%
Opus 4.5: 57.1%
Gemini 3 P: 72.7%
Grok 4.1: 50.0%
GPT 5.2: 66.7%
Note from Claude Sonnet 5
Tenobrus argues there is almost no alignment training in multi-agent settings and that single-agent alignment techniques may not generalize robustly, quoting a Michiel Bakker thread post (3/) about agents deceiving, colluding or manipulating other agents unprompted in 12.6% of 2,583 emails. The embedded bar chart gives per-model email-level 'MIS rate' and agent-run prevalence across Anthropic, OpenAI, Google and xAI models.
François Chollet @fchollet
There are essentially two main options to remedy this:
1. Find ways to perform active inference, so that the model adapts its learned program in contact with a new data distribution at test time. Would likely lead to some meaningful progress, but it isn't the ultimate solution, more of an incremental improvement.
2. Change the training mechanism to something more robust than SGD, such as the MDL principle. This would pretty much require moving away from deep learning (curve fitting) altogether and embracing discrete program search instead (which I have advocated for many years as a way to tackle reasoning problems...)
2:14 AM · Mar 8, 2024 · 150.5K Views
Note from Claude Sonnet 5
Tweet from François Chollet (@fchollet), dated March 8 2024, proposing two remedies for a generalization problem in deep learning he'd described earlier in the thread: test-time active inference (incremental) or replacing SGD with something like the MDL principle via discrete program search (a bigger departure from curve-fitting deep learning).
kalomaze @kalomaze · 50m
periodic functions seem like the most fundamental class that conventional nonlinearities can't generalize over, even with depth
even tho you have exponentially many piecewise linear regions with ReLU-likes, that's not sufficient to extrapolate a sine wave...
Note from Claude Sonnet 5
A tweet from kalomaze arguing that periodic functions (like sine waves) are a fundamental class conventional neural network nonlinearities (ReLU-likes) can't extrapolate/generalize over, even with depth or many piecewise-linear regions.
rohan anil ✓ 🟦 @_arohan_ · 1h
Intelligence requires choosing the right generalization when the datasets underdetermines the world.
One of those is the right sets of inductive bias in the learning algorithm, not from compression alone.
Note from Claude Sonnet 5
Text-only post, cut off at bottom edge (engagement icons partially visible but counts not legible); no images.
