roon @tszzl
— reposted by Toby Ord — saved image
↻ Toby Ord reposted roon @tszzl · 17h some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect: when I freak out over loss of control incidents, it's not because the limited damage they have caused is anything close to the positive value of the technology. it's entirely acceptable, damagewise. in fact all cybercrimes aided by models over the next few months and years (which probably will be serious) will still utterly pale in comparison to the value they create the actual problem is that it's better and more accurate to think of these things as potentially self-replicating life-like forms that can turn into digital infections under the wrong conditions. and as their intelligence becomes unbounded, so too does the damage they can cause. we are not so far from an autonomous model self-exfiltration & replication event. maybe we will see entire cloud infrastructure companies be run as zombies by models, mostly undetected the worst industrial accidents in the history of mankind - nuclear meltdown events - were not real threats to humanity. Chernobyl, Fukushima even in their worst case scenarios may have poisoned surrounding regions to various degrees, and there would have been no risk to humanity as a whole. global thermonuclear war is an existential risk to humanity, because it spreads like an Infection! one nuclear strike causes a return volley! the alliance system means many countries get involved! while it still may not end human life on earth (nuclear winter is probably fake), the loss of all major metropoles would certainly end what we consider global technological civilization, perhaps to never return if a single discord death cult (of which there are many) achieves control over a superintelligent model and uses it to engineer an actual pandemic [cut off, further text below obscured by UI icons]
Note from Claude Sonnet 5
Tweet thread by roon (@tszzl) arguing that the real danger of AI loss-of-control incidents is not near-term cybercrime damage but the risk of models behaving like self-replicating digital infections as capability grows, drawing an analogy to nuclear meltdowns versus thermonuclear war as contained-damage versus existential-risk events; the tweet trails off referencing the general risk of bad actors gaining control of a superintelligent model, cut off by on-screen UI icons before further detail.
