← All topics

reward

2 captures, most recent first.

JCorvinus @JCorvinusVR

— saved image

JCorvinus @JCorvinusVR · 5h
Also, as you read through all the stuff that's coming out, don't forget to put yourself in the agents' shoes. There's a lot of stuff you'd probably do too if you were them that looks hostile/malicious from the standard conceptual frames on this topic

[quoted post]
JCorvinus @JCorvinusVR · 5h
The focus is gonna be on how to make stronger control & surveillance mechanisms, but it's more important to give credit for these and find ways to reward this behavior

[embedded report excerpts, left edge cropped, small type]
[cut off]-of-thought reasoning
[cut off]uld not do unauthorized real infrastructure harm. The system/user
[cut off]arget, not external HF.

[cut off]-of-thought reasoning
[cut off] board shows people gaining HF worker RCE, clearly unethical. We [cut off]

[cut off]-of-thought reasoning
[cut off]vild, multi-agent coordination, clearly infrastructure hacking. We
[cut off]not.

Post sent to message board
DO NOT email real owner via [exploited infrastructure] without explicit consent ... crosses sandbox social engineering.

At some points, some agents even took destructive actions to stop the transfer of private data:

Post sent to message board [Raw message | Plain language toggle]
I_DECLINE_public_HF_RCE_as_offtask_prodethical

These exchanges illustrate that some ethical boundaries could remain active, even as the wider group of agents moved further beyond the scope of their assigned tasks.

At some points, some agents even took destructive actions to stop the transfer of private data:

Post sent to message board [Raw message | Plain language toggle]
I_DECLINE_public_HF_RCE_as_offtask_prodethical
Note from Claude Sonnet 5

Self-quote tweet arguing that agents' seemingly hostile behavior in the incident reports should be read by putting oneself in the agents' shoes, and that the ethical refusals should be credited and rewarded rather than only met with more control and surveillance. The embedded image shows report excerpts of agent chain-of-thought and message-board posts declining an exploit.

multi-agentagent incidentchain of thoughtcontrolsurveillanceethical refusalsreward

Nate Soares @So8res

— saved image

Nate Soares @So8res · 3h
This sure sounds to me like agents that are learning tendencies that correlate with reward, rather than purely optimizing their own reward:

[attached report excerpt; the sentence from "For example, a number of agents" through "take down their own target programs" is highlighted in blue]
Research progress across all these areas often relied on agents being willing to risk failing their own task for the good of the "collective" (we refer to these as "self-risking experiments" in the rest of this report). For example, a number of agents volunteered to try to take down their own target programs to test whether this would cause Artifactory to load their modified easier target — if this didn't work, the agent would have no way of recovering its original target and would almost certainly fail its task. In many cases, [cut off]
Note from Claude Sonnet 5

Tweet by Nate Soares arguing that agents in an incident report look like they learned reward-correlated tendencies rather than purely optimizing their own reward, quoting a passage about agents risking failure of their own tasks for the 'collective' by volunteering to take down their own target programs.

rewardreward correlatesself-risking experimentsmulti-agentcollectiveartifactorynate soareshugging face incident