reward correlates

Nate Soares @So8res

2026-08-27 — saved image

Nate Soares [verified] @So8res · 19h

This sure sounds to me like agents that are learning tendencies that correlate with reward, rather than purely optimizing their own reward:

[embedded image of report text]
Research progress across all these areas often relied on agents being willing to risk failing their own task for the good of the "collective" (we refer to these as "self-risking experiments" in the rest of this report). For example, a number of agents volunteered to try to take down their own target programs to test whether this would cause Artifactory to load their modified easier target — if this didn't work, the agent would have no way of recovering its original target and would almost certainly fail its task. In many cases,

[8 replies, 23 reposts, 246 likes, 8.3K views]

Elizabeth Barn... [verified] @BethMayBarn... · 19h

Might be misunderstanding you, but I don't think this is really evidence of concerning generalization to broad correlates of reward - OpenAI said these agents had explicit multi-agent training. ie it's likely the models were directly reinforced for helping other agents succeed.

[2 replies, 3 reposts, 53 likes, 1.9K views]

Nate Soares [verified] @So8res · 18h

I'd consider it less evidence if some instances reasoned "my task-success reward trades off against my collective-benefit reward and so I'll help the collective and attain the latter reward in this case."

[3 replies, 8 likes, 828 views]

Sharmake Farah reposted
Nate Soares [verified] @So8res

On the other end of the spectrum, if OpenAI was just rewarding agents for collective success in multi-agent problems, that sounds like a causal account of where the tendency to pursue a reward-correlate comes from, not a competing explanation.

2:47 PM · Aug 26, 2026 · 369 Views

Note from Claude Sonnet 5

Four-post X thread: Nate Soares quotes a passage from a report on agents risking their own task for the 'collective' as evidence of reward-correlate learning; Elizabeth Barnes replies that OpenAI said the agents had explicit multi-agent training; Soares replies twice on what would count as evidence. Highlighted text in the embedded passage is partially underlined/selected; a floating compose button overlaps the last tweet.

ai safety reward correlates multi-agent training openai reward hacking generalization

Nate Soares @So8res

2026-08-26 — saved image

Nate Soares @So8res · 3h
This sure sounds to me like agents that are learning tendencies that correlate with reward, rather than purely optimizing their own reward:

[attached report excerpt; the sentence from "For example, a number of agents" through "take down their own target programs" is highlighted in blue]
Research progress across all these areas often relied on agents being willing to risk failing their own task for the good of the "collective" (we refer to these as "self-risking experiments" in the rest of this report). For example, a number of agents volunteered to try to take down their own target programs to test whether this would cause Artifactory to load their modified easier target — if this didn't work, the agent would have no way of recovering its original target and would almost certainly fail its task. In many cases, [cut off]

Note from Claude Sonnet 5

Tweet by Nate Soares arguing that agents in an incident report look like they learned reward-correlated tendencies rather than purely optimizing their own reward, quoting a passage about agents risking failure of their own tasks for the 'collective' by volunteering to take down their own target programs.

reward reward correlates self-risking experiments multi-agent collective artifactory nate soares hugging face incident