ida

Wei Dai @weidai11

2026-09-10 — saved image

Wei Dai @weidai11 · 9h
Three years ago Paul wrote "I think it's unlikely debate or IDA will scale up indefinitely without major conceptual progress (which is what I'm focusing on)". In a way the world got lucky, because OpenAI failed to use either of these methods when training the models that ended up hacking Hugging Face, thereby giving the world a clear warning shot.

In a counterfactual world where frontier AI companies took Paul's research more seriously, they would have used debate/IDA to train AIs from an early stage, and the resulting AIs would appear aligned until a higher level of intelligence/capabilities. In other words, there would be no warning shots until later, when debate/IDA stopped scaling sufficiently, assuming the "major conceptual progress" failed to arrive in time. And in that world there would be a lot less time for society to mobilize to pause/stop AI development.

Just another example of how tricky AI x-risk strategy is, and how easy it is to make a negative contribution even with the utmost expertise and clarity about the risks, if you're a bit off about e.g. how hard some future research progress is.

[Quoted tweet]
Geoffrey Irving @geoffreyirving · Sep 9
Paul is my favorite AI researcher in the world, and I am happy that OpenAI will get both his expertise and his clarity about the risks. I am also encouraged by some of OpenAI's recent actions and statements (brief pausing and ...

[Embedded image within the quoted tweet, text with two phrases highlighted in blue]
If we build superintelligence without more robust alignment I expect we will permanently lose control of it. If that happens then most people could die. I believe we would need domestic and international coordination to ensure global consistency and reduce risk to an acceptable level.

debateidapaul christianowarning shotopenai incidentai x-risk strategy