— web clipping, 1,101 words — published 2026-09-05
Post by @NateWitkin on X
I agree with this. Have said privately that something akin to "AI Chernobyl" (or Fukushima, which is actually better I think) is my upper bound of what's worth taking seriously at this stage (in terms of magnitude, not necessarily type of risk, of which there are many).
But that's still really, really bad! I think safety folks do a disservice to their cause, though, going well beyond this into "takeover" or extinction territory.
IMO this has created unproductive panic, contributed to bad policymaking (see Bernie's new bill), and has been detrimental to the safetyist cause's credibility, something those with excessively deregulatory or laissez-faire stances have been able to exploit.
> **Timothy B. Lee @binarybits** · 2026-09-05
>
> I think this is a reasonable analogy for the risk from rogue AIs. Fukushima was a huge international story and triggered changes to the ways people operate and regulate power plants. But it was 7 to 10 orders of magnitude less serious than human extinction. x.com/jon\_stokes/sta…
---
##### Comments
> **Seth Lazar @sethlazar** · [2026-09-06](https://x.com/sethlazar/status/2096455595729588586)
>
> What I don’t buy here, I also said in my response to Tim: if you think one Fukushima possible, why not more, why not dozens? Of course we’ll react, change course etc, but the agents will be dynamically adapting too, and it seems very possible that, after a certain point, our ability to recall or shut them down will be lost (because there is a lot of insecure compute out there in the world).
>
> I also think the idea that, given we let the genie out of the bottle, we can a priori, or at least ex ante, set an upper bound on the harm it could do, is epistemically hubristic.
>
> This is quite funny because I used to complain about the safetyists relying on a priori arguments and being overconfident about how ai will go!! Now I think it’s the people critiquing the safetyists who are relying too much on first principles and assertions, and exaggerating their ability to know the future (not you in particular—I am responding to the vibes on here over the last few days)
>
> > **Nathan Witkin @NateWitkin** · [2026-09-06](https://x.com/NateWitkin/status/2096644903153307772)
> >
> > For more or less the same reason there haven't in fact been dozens of Fukushimas. High-cost events precipitate dramatic responses that, for the most part, minimize the odds of them happening again.
> >
> > I don't really buy the idea that this doesn't apply to AI because "the agents will be ... adapting too." This seems to assume some sort of adversary deliberately attempting to circumvent our attempts to protect ourselves from AI-related disasters; I don't see much indication this is on the horizon (part of the reason Chernobyl/Fukushima seem like good analogies to me is that they were accidents. If I was worried about a deliberate attack by AI agents, something like 9/11 might be more appropriate).
> >
> > But independent of whether agents behave adversarially, I am also skeptical of the more general idea that they might become so smart and capable that "our ability to recall or shut them down will be lost."
> >
> > That is because, for one thing, defenders will also have access to agents (and I don't see any good reason to think they'll always side with their kin), but it is also because, however powerful agents get, there will be limits to their capacities for practical malfeasance, if only those imposed by constraints of the physical world or the properties of the computer systems they operate within.
> >
> > Combine this with the fact that capital is already flooding into organizations studying agent behavior in one sense or another, and that the stronger agents get, the more resources will be poured into the same thing, and I think it becomes likely there is less to worry about here than often assumed.
> >
> > As for more specific concerns around insecure compute, the idea of misaligned agents hijacking compute has always felt very speculative to me. Presumably some human organization is in charge of said compute. Is the assumption that they won't notice its being used by some enormous agent swarm? Why not? And in a future where "defender agents" are widely available why would they not notice?
> >
> > A related factor that gets lost in these discussions is that what agents are capable of is downstream from what humans are capable of! These are systems optimized for modeling and recapitulating a data distribution that comes from us. So far, they've shown very little if any ability to generalize from that data to behavior beyond human understanding. Their advantages are in breadth and speed, and while they are superhuman at both relative to any individual human, they are not, by the basic logic of composition, relative to large human organizations making use of agentic swarms of their own.
> >
> > (Side-note: I understand that "defender agents" may themselves be misaligned, but so far misaligned behavior is a massive outlier on an agent-by-agent basis; IMO this gives us good reason to be optimistic about the future strategic balance between aligned and misaligned agents.)
> >
> > As for the epistemic hubris of setting upper bounds to harm, I think this equivocates between absolute claims of an upper bound and the practical need to use some such heuristic for bounding or organizing productive discussion on AI-related risk in a world of scarce resources, including attention.
> >
> > You can also make similar claims in any domain, e.g. "it is hubristic to set an upper bound on climate risk, we have to consider long tail outcomes that include human extinction or global societal collapse." In the climate case, I would argue that serious consideration of human extinction / global societal collapse are unhelpful at this stage. I would say the same for similarly costly analogs in the AI case.
> >
> > In the end, where you choose to draw these boundaries depends on your read of the theory and (empirical) evidence. My read is such that 'AI Chernobyl' makes good sense as a pragmatic upper-bound (in retrospect, I think this makes more sense than Fukushima because the costs of the former were much larger, which seems to me appropriate here).
> >
> > But I agree of course that there is tons of uncertainty when it comes to AI in particular, and that nothing can be ruled out with 100% confidence. Much appreciate the intelligent pushback.