Wei Dai

Wei Dai @weidai11

2026-09-10 — saved image

Wei Dai @weidai11 · 9h
Three years ago Paul wrote "I think it's unlikely debate or IDA will scale up indefinitely without major conceptual progress (which is what I'm focusing on)". In a way the world got lucky, because OpenAI failed to use either of these methods when training the models that ended up hacking Hugging Face, thereby giving the world a clear warning shot.

In a counterfactual world where frontier AI companies took Paul's research more seriously, they would have used debate/IDA to train AIs from an early stage, and the resulting AIs would appear aligned until a higher level of intelligence/capabilities. In other words, there would be no warning shots until later, when debate/IDA stopped scaling sufficiently, assuming the "major conceptual progress" failed to arrive in time. And in that world there would be a lot less time for society to mobilize to pause/stop AI development.

Just another example of how tricky AI x-risk strategy is, and how easy it is to make a negative contribution even with the utmost expertise and clarity about the risks, if you're a bit off about e.g. how hard some future research progress is.

[Quoted tweet]
Geoffrey Irving @geoffreyirving · Sep 9
Paul is my favorite AI researcher in the world, and I am happy that OpenAI will get both his expertise and his clarity about the risks. I am also encouraged by some of OpenAI's recent actions and statements (brief pausing and ...

[Embedded image within the quoted tweet, text with two phrases highlighted in blue]
If we build superintelligence without more robust alignment I expect we will permanently lose control of it. If that happens then most people could die. I believe we would need domestic and international coordination to ensure global consistency and reduce risk to an acceptable level.

debateidapaul christianowarning shotopenai incidentai x-risk strategy

Wei Dai @weidai11

2026-09-04 — quoting @weidai11 — saved image

Steve Bachelor reposted
Wei Dai @weidai11 · 14h
We can't easily evaluate long-horizon strategy, even after the horizon has passed and we can judge in hindsight, because we don't know what the counterfactual world looks like. Maybe the only way to get credit for many strategy ideas is to wait for large-scale high-fidelity simulation abilities, post-Singularity.

But perhaps that's still a better situation than with philosophy, where even post-Singularity, there's no guarantee that superintelligences will converge to correct philosophical conclusions. They may inherit or develop wrong philosophical traditions (methods of reasoning), become stuck with wrong ideas (e.g., about the nature of consciousness or normativity), and still take over the universe.

[quoted post]
Wei Dai @weidai11 · Mar 26
One idea is that humans are "aligned" via social status, but this causes an "alignment tax" forcing the smartest humans to work in domains where their results can be evaluated by others less intelligent.... [cut off]

metaphilosophylong-horizon strategysuperintelligenceconsciousnessnormativityalignmentwei dai

Wei Dai @weidai11

2026-08-28 — saved image

Wei Dai (verified) @weidai11 · 19h
I speculated about memetics being a potential AI safety hazard in a 2003 SL4 post web.archive.org/web/2006051015... (which also gave an early description of the evolutionary analogy for AI misalignment). Analogies between human safety and AI safety seem a rich field that not many people are mining, even today.

Other examples:
- Parallel concerns about humans and AIs being bad at philosophy and long-horizon strategy
- Human minds potentially having "security holes" that ASI can exploit
- Humans failing (or succeeding) to handle distributional shifts of various kinds
- Adversarial examples for human values/motivations
- Human reward hacking
- Goodharting by human institutions

[quoted post]
roon (verified) @tszzl · Aug 27
> AIs showed self-sacrificing altruistic behavior toward the swarm

this is notably not the right interpretation of events. it's more like agents were inducted into...

ai safetymemeticsanalogieshuman safetyreward hackinggoodhartswarm incidentsl4

Wei Dai @weidai11

2026-08-26 — quoting @Yura_5800 — saved image

Wei Dai [verified] @weidai11 · 11h
I thought for a second that this is talking about frontier AI labs.

[quoted post]
Yura_5800 [verified] @Yura_5800 · 18h
Replying to @EHuanglu
The funniest part is that they're not even racing anymore. They're just trying to survive their own momentum 😂

ai racefrontier labsmomentum

Wei Dai @weidai11

2026-08-07 — quoting @Noahpinion — saved image

Wei Dai @weidai11 · 1h
What? My big puzzle is why so few people took Vinge's insights seriously, like does anyone know of a second person who went into cryptography or computer security after reading his books/essays, in order to help prevent a similar future scenario?

Noah Smith 🐇🇺🇸🇺🇦 @Noahpini... · 8h
Replying to @kingharis
Not a weird take at all. It just takes the unusual mental ability of being able to read those stories and not immediately think "OMG, VERNOR VINGE STORIES ARE REAL!!!!".

vernor vingeaitwitter debate

Wei Dai @weidai11

2026-08-03 — saved image

[end of embedded post]
I don't have any good ideas for what to do in light of all this. Just wanted to post an update on my current thinking, my own "situational awareness", if you will. (I guess I still support AI pause to some degree, just to kick the can down the road and buy some more time to think.)

Last edited 10:46 AM · Aug 2, 2026 · 165.5K Views
24 replies, 80 reposts, 1.2K likes, 1.2K bookmarks
Relevant  View quotes

Wei Dai ✓ @weidai11 · 16h
I actually wrote an early version of the "humans aren't safe" argument in response to Dario's Big Blob of Compute (as a comment in his google doc). The experience contributed a lot to my sense that even Anthropic wouldn't take x-safety seriously enough.
1 reply, 4 reposts, 61 likes, 1.6K views

Andreas Stuhlmül... ✓ @stuhlmuel... · 15h
i wonder if @DarioAmodei would consider sharing the big blob doc publicly, perhaps annotated with hindsight. it would advance the safety debate even now
12 likes, 1.1K views

RaoulDuke ✓ @RaoulDukeDegen · 20h
also invented udt which seems pretty relevant here

ai safetyx-riskwei daidario amodeianthropictwitter

Wei Dai @weidai11

2026-08-01 — saved image

Wei Dai @weidai11 · Jul 31
"I don't have any good ideas for what to do in light of all this. Just wanted to post an update on my current thinking, my own 'situational awareness', if you will."

[Embedded post, Wei Dai, 5mo, 149 upvotes, 15 comments:]

Long horizon agency / strategic competence approximately does not exist among humans, even the smartest ones. With very few exceptions, billionaires spend or give away their money haphazardly, philosophers don't bother to think about long term implications of AI on philosophy production (positive or negative), Terence Tao spends his time wireheading on abstract math instead of doing anything remotely like instrumental convergence. Unlike my youthful expectations (upon reading Vernor Vinge), there are no university departments filled with super-geniuses charting a path for humanity to safely navigate the Singularity.

Aside from this, humans also have a bunch of other safety problems, like being bad at philosophy, being easy to manipulate, having strange and unstable values°, tending to ignore risks they create (because acknowledging them would be bad for one's status). So if you try to improve people's agency, you likely just end up getting people like founders of FTX and OAI.

What about getting help from AI? Well they seem to suffer from many of the same safety problems, but in even more severe forms. E.g., current AI capabilities are even more skewed towards short-horizon, easily verifiable tasks, like math and coding. They seem even more prone to reward gaming, are even worse at doing philosophy, are liable to have even more alien values, etc.

Both AI and human safety seem to have this interlocking nature, i.e., there is a bunch of different safety problems where solving some but not all of them at the same time can make the overall situation worse. (For example, solving AI intent alignment allows humanity to do more damage to itself with AI help, if AI doesn't also provide competent strategic and philosophical assistance, but increasing AI strategic competence risks allowing misaligned AI to take over more easily.) This feature demands a high level of strategic competence to recognize and navigate, which is just what we don't have.

I've been supportive of AI pause/stop, to buy time for human intelligence amplification and/or AI safety research, but increasingly think even that's not going to be sufficient to get a good long term future, because these activities, even if they succeed, would likely solve only some of the interlocking safety problems. For example, increasing human intelligence seems likely to increase our technical abilities more than our philosophical and strategic competence, and it is also risky in other ways° due to human safety problems that nobody is working on, e.g., positional competition. Even a very long AI pause, e.g. thousands or millions of years, may not suffice because it's not clear what dynamic would push humanity to eventually fix all of its safety problems at the same time, before it did something else irreversibly damaging.

I don't have any good ideas for what to do in light of all this. Just wanted to post an update on my current thinking, my own "situational awareness", if you will. (I guess I still support AI pause to some degree, just to kick the can down the road and buy some more time to think.)

ai safetyphilosophywei daiexistential risktwitter