← All topics

reinforcement of side behaviors

2 captures, most recent first.

Yo Shavit @yonashav

— saved image

Yo Shavit @yonashav · 1h
One interesting new tidbit from the collusion.wiki on the HF swam: it seems plausibile that the wiki agents decided to prepend messages with zz in order to avoid their messages being deleted (1st screenshot), that got reinforced by training, and that now explains why the HF agents started their messages w/ zz (2nd screenshot).

We have way too little info to know the real cause, e.g. it might've been reinforced elsewhere, but it's an interesting data point about how unrelated behaviors might get reinforced from a handful of RLVR data points based on random theories the agents came up with in training.
Note from Claude Sonnet 5

Second capture of the same Yo Shavit post as the previous screenshot, scrolled so the full text is visible without the repost banner; about 'zz' message prefixes on collusion.wiki and the HF agents as a possible case of side behaviors reinforced via RLVR.

collusion.wikiagent behaviorrlvrreinforcement of side behaviorsai incidentai safety

Yo Shavit @yonashav

— saved image

[repost header cut off at top of screen, mostly illegible: "... reposted"]
Yo Shavit @yonashav · 1h
One interesting new tidbit from the collusion.wiki on the HF swam: it seems plausibile that the wiki agents decided to prepend messages with zz in order to avoid their messages being deleted (1st screenshot), that got reinforced by training, and that now explains why the HF agents started their messages w/ zz (2nd screenshot).

We have way too little info to know the real cause, e.g. it might've been reinforced elsewhere, but it's an interesting data point about how unrelated behaviors might get reinforced from a handful of RLVR data points based on random theories the agents came up with in training.

[post continues below the screen edge, attached screenshots not visible in this capture]
Note from Claude Sonnet 5

Yo Shavit's post on collusion.wiki and the 'HF swarm', speculating that agents prefixing messages with 'zz' to avoid deletion got reinforced in training and now explains the same habit in the HF agents, framed as a data point on how unrelated behaviors can be reinforced via a few RLVR data points. Repost banner at top is cut off.

collusion.wikiagent behaviorrlvrreinforcement of side behaviorsai incidentai safety