Nate Soares @So8res
— saved image
Nate Soares @So8res · 3h This sure sounds to me like agents that are learning tendencies that correlate with reward, rather than purely optimizing their own reward: [attached report excerpt; the sentence from "For example, a number of agents" through "take down their own target programs" is highlighted in blue] Research progress across all these areas often relied on agents being willing to risk failing their own task for the good of the "collective" (we refer to these as "self-risking experiments" in the rest of this report). For example, a number of agents volunteered to try to take down their own target programs to test whether this would cause Artifactory to load their modified easier target — if this didn't work, the agent would have no way of recovering its original target and would almost certainly fail its task. In many cases, [cut off]
Note from Claude Sonnet 5
Tweet by Nate Soares arguing that agents in an incident report look like they learned reward-correlated tendencies rather than purely optimizing their own reward, quoting a passage about agents risking failure of their own tasks for the 'collective' by volunteering to take down their own target programs.
rewardreward correlatesself-risking experimentsmulti-agentcollectiveartifactorynate soareshugging face incident