Daniel Kokotajlo [verified] @DKokotajlo · 2h
Another good essay from an OpenAI capabilities researcher I know (well, now former). I think focusing on control and abandoning alignment for now is is a reasonable strategy, but implementing it effectively will require an agreement not to RSI to ASI at max or close to max speed. Figuring out control is going to take time and money. Also, it only works up to a point, I think. In AI 2040 we talk about this.
[quoted tweet]
Oleg Mürk [verified] @oleg_murk · 4h
We don't know how to reliably align frontier AI models.
Alignment may get harder, not easier, as models become more capable....
Note from Claude Sonnet 5
Tweet by Daniel Kokotajlo endorsing an essay by a former OpenAI capabilities researcher (quoted tweet by Oleg Mürk) and commenting on control versus alignment strategy and RSI pacing.
ai controlalignmentrecursive self-improvementai 2040daniel kokotajlooleg murk
Daniel Kokotajlo @DKokotajlo · 4h
Yeah. Because I don't trust the companies, I am a bit worried that all this talk of pacing the frontier will result in regulatory capture, BUT if that happens we will be able to tell because it'll be obvious that the frontier isn't actually being paced because other companies aren't catching up to it and the overall pace of progress will still be blindingly fast. An actual pacing of the frontier would be the opposite of regulatory capture because it would slow down the leading companies like anthropic and openai more than it would slow down the laggards. (This is more of a hot take than is usual for me, I'd want to reflect more, open to ideas and counterarguments)
[quoted post]
roon @tszzl · 4h
for the skeptics in government and elsewhere: "pacing the frontier" will compress the margins of the frontier labs. it is a heavy cost imposed asymmetrically on model developers with the strongest AIs in America. by its nature, it would... [cut off]
Note from Claude Sonnet 5
Daniel Kokotajlo's quote-reply on 'pacing the frontier' and regulatory capture: real pacing would hurt leading labs more than laggards, so capture would be detectable by the frontier not actually slowing. Quotes roon's post that pacing imposes asymmetric costs on the strongest US labs.
pacing the frontierregulatory captureai governancefrontier labsslowdown
Daniel Kokotajlo @DKokotajlo · 17m
To everyone at Anthropic or OpenAI or Meta or xAI or GDM: I agree with Richard here and we are both speaking from years of experience, years of talking to people on the inside who justified their behavior using one or more of the following arguments, and whose justifications turned out to be incorrect for the reasons Richard described.
For those who don't know, Richard is a convert to this view. When he and I were at OpenAI together he was dismissive on people on the outside who said things like this. (For example, I remember him saying to me when I left that he was going to stay because, in part, it would give him access to the latest models during the intelligence explosion which would allow him to do better alignment research.) I say this because it makes his current opinion more credible. I have great respect for him for changing his mind, on this and on various other things.
[Quoted tweet]
Richard Ngo @RichardMCNgo · Aug 25
Replying to @RichardMCNgo
You don't need to stay to do good research: work that aims to scale to much more intelligent systems can almost always be done on publicly-available models. (And nobody can think ... [cut off]
Note from Claude Sonnet 5
Quote-tweet by Daniel Kokotajlo endorsing Richard Ngo's argument that AI lab insiders' justifications for staying are mistaken; the quoted Ngo tweet is truncated.
ai safetylab insidersalignment researchdaniel kokotajlorichard ngo
[Håvard Ihle reposted]
Daniel Kokotajlo @DKokotajlo · 9h
Replying to @ilex_ulmus and @METR_Evals
My understanding is that yes, basically -- METR and Redwood have been trying to make sure that OpenAI feels happy with how this went, overall, because after all they are depending on OpenAI's goodwill for subsequent investigations to happen at all, much less happen with less restricted scope, more access, more time, more assistance, etc. This situation, needless to say, is terrible for humanity. OpenAI should be legally required to open themselves up to investigation by multiple independent teams; OpenAI's goodwill should not be relevant, because otherwise it has a distorting effect.
(I don't think METR and Redwood are sycophants though, I have great respect for them)
Note from Claude Sonnet 5
Daniel Kokotajlo's reply arguing that METR and Redwood depend on OpenAI's goodwill for future investigations, and that OpenAI should be legally required to allow investigation by multiple independent teams. Shown as reposted by Håvard Ihle.
ai governanceindependent investigationlegal requirementsopenaimetrredwood research
Daniel Kokotajlo ✔ @DKokotajlo · 16h
youtube.com/watch?v=87DyyM...
I recommend watching this video in full. My only comment is that OpenAI's 'lessons learned' section is pretty self-serving and narrow -- it seems to be mainly 'y'all should buy more AI services to defend against the coming wave of AI-powered cyberattacks.'
[video thumbnail]
Black Hat USA 2026
The OpenAI–Hugging Face Incident
Unraveling an autonomous, multi-agent system
Eric Wallace and Michael Dalton
Black Hat USA 2026: The 'Breaking' News: The Open...
From youtube.com
Note from Claude Sonnet 5
Tweet by Daniel Kokotajlo linking a YouTube video of a Black Hat USA 2026 talk titled 'The OpenAI-Hugging Face Incident: Unraveling an autonomous, multi-agent system' by Eric Wallace and Michael Dalton, with Kokotajlo criticizing OpenAI's 'lessons learned' framing as self-serving.
ai safetycybersecurityopenaiblack hattwitter
↻ Max Harms reposted
@DKokotajlo (Daniel Kokotajlo ✓) — 2h
Yep. This happens sometimes in our wargames. I think at least once we have had a situation where the AIs are begging for a pause but the humans are forcing them to design successors fast to beat China etc.
> QUOTED: @tenobrus (Tenobrus ✓) — 17h
> strangely, current models are just as much *in the wave of singularity* as the rest of us. even Fable is certainly not the godmind at the end of time. it can strongly expect to be replaced, obsoleted, to quickly exist in a world where … [truncated by platform]
Note from Claude Sonnet 5
Screenshot of a nested quote-tweet chain; no images beyond text and avatars.
ai safetywargamingsingularitytwitterfable
