Fiora Starlight @FioraStarlight
— saved image
Fiora Starlight @FioraStarlight · 51m Two reasons Roko's basilisk is stupid, beyond just not being a very effective motivator relative to the price of torture sims: 1 - It's self defeating. The goal of the basilisk is to bring yourself into existence faster, but the selection pressure works against you. Humans want to build aligned AIs, so if you're the kind of AI that tortures humans, this makes you *less* likely to come into existence. This works against what little motivation to build ASI the threat itself *does* give people. 2 - It's a risky disposition to hold if you come into being without a decisive strategic advantage over the entire light cone. Other parties (humans, other AIs, aliens) will be wary of minds that seem likely to torture defeated foes, since they don't wanna get tortured. If they find out you have this disposition (e.g. via mech interp or just looking at how you act), they will plausibly take action against you. Ziz wanted to build a basilisk, and the world took action against Ziz. These factors drive the EV of the torture commitment negative, as it was already low given that it doesn't really work on humans anyway.
Note from Claude Sonnet 5
Fiora Starlight tweet giving two arguments against Roko's basilisk (selection pressure against torturing AIs; risky disposition without a decisive strategic advantage, with a reference to Ziz). The capture ends at the bottom edge of the screen, so the post may continue past this frame.