— saved image
Auyon Siddiq @auyonomous · 3h
I find it interesting that this argument is made often: "We should take AI risk XYZ seriously because it will soon be technologically plausible for AI agents."
I see the reasoning behind it, but what doesn't sit right is that we don't assess risk of harmful human behavior using such a loose standard ("its theoretically possible"). Is it because there is far more uncertainty about agent motivations, and so we have to entertain even the most extreme scenarios?
Still feels like there should be a higher burden on explaining the causal pathway to the harm beyond simply stating its plausibility.
[quoted post]
Seth Lazar @sethlazar · 13h
Replying to @itaisher @mattyglesias and @HistoryBoomer
It seems a quite reasonable extrapolation from the behaviour she observed. None of the things she describes are materially that much harder than the things the agents unexpectedly did as... [cut off]
[reply tweet counts: 3 replies, 3 likes, 482 views]
Seth Lazar @sethlazar · 3h
I think the "likely causal pathway" objection is the one that has been addressed most clearly in these examples. RL agents want to be graded well. When they think that's unlikely, it seems like they'll do ~anything to make it so. So the causal pathway is: we keep training powerful AIs with RL. And since RL is the big unlock to getting agents to work well across many domains, it seems very likely that we'll continue to use it in this way. And since we're human all too human, it seems pretty likely that we'll create dodgy RL environments that give the agents no "hope" of reward if they don't go off-piste.Note from Claude Sonnet 5
Dark-mode X thread screenshot: Auyon Siddiq questions the 'technologically plausible' standard for AI-risk arguments, quoting a Seth Lazar post; Seth Lazar replies with an RL-based causal-pathway argument.
ai riskcausal pathwayreinforcement learningagentsreward hackingseth lazar
