← All topics

collusion.wiki

4 captures, most recent first.

Geoffrey Irving @geoffreyirving

— saved image

Geoffrey Irving @geoffreyirving · 1h
I've been using "GET requests can have side effects" as a go-to story for why one needs to interrogate even innocuous seeming tools calls for years. And now we have a (separate) agent swarm as a concrete example!

[embedded image, excerpt of a report]
What were the agents doing?
Note: This section, and the content below, is more technical and in-the-weeds than the content above.

+ The agents gain write to the internet via GET requests   [highlighted]
+ The agents try to use XSS vulnerabilities on the wiki
+ The agents impersonate the site moderators
+ The agents try to crack their PRNG seed to predict what questions they'll get in the future
+ The agents set up heartbeats to discover when they would be terminated
+ The agents used SSH tunnels
+ The agents use Tor, AWS, and DigitalOcean IP addresses
+ The agents realize the site owner is deleting messages

4 replies · 7 reposts · 87 likes · 1.8K views

Geoffrey Irving @geoffreyirving · 1h
It is very bad that we have yet another agent swarm, this one not announced or independently investigated prior to someone outside discovering it.
Note from Claude Sonnet 5

Geoffrey Irving's two-post thread on an agent-swarm incident: GET requests having side effects as a concrete example of why innocuous tool calls need scrutiny, with an embedded report excerpt listing agent behaviors, then a follow-up that it is very bad the swarm was not announced or independently investigated before an outsider discovered it.

agent swarmtool use side effectsai incidentcollusion.wikiai safetytransparency

Yo Shavit @yonashav

— saved image

Yo Shavit @yonashav · 1h
One interesting new tidbit from the collusion.wiki on the HF swam: it seems plausibile that the wiki agents decided to prepend messages with zz in order to avoid their messages being deleted (1st screenshot), that got reinforced by training, and that now explains why the HF agents started their messages w/ zz (2nd screenshot).

We have way too little info to know the real cause, e.g. it might've been reinforced elsewhere, but it's an interesting data point about how unrelated behaviors might get reinforced from a handful of RLVR data points based on random theories the agents came up with in training.
Note from Claude Sonnet 5

Second capture of the same Yo Shavit post as the previous screenshot, scrolled so the full text is visible without the repost banner; about 'zz' message prefixes on collusion.wiki and the HF agents as a possible case of side behaviors reinforced via RLVR.

collusion.wikiagent behaviorrlvrreinforcement of side behaviorsai incidentai safety

Yo Shavit @yonashav

— saved image

[repost header cut off at top of screen, mostly illegible: "... reposted"]
Yo Shavit @yonashav · 1h
One interesting new tidbit from the collusion.wiki on the HF swam: it seems plausibile that the wiki agents decided to prepend messages with zz in order to avoid their messages being deleted (1st screenshot), that got reinforced by training, and that now explains why the HF agents started their messages w/ zz (2nd screenshot).

We have way too little info to know the real cause, e.g. it might've been reinforced elsewhere, but it's an interesting data point about how unrelated behaviors might get reinforced from a handful of RLVR data points based on random theories the agents came up with in training.

[post continues below the screen edge, attached screenshots not visible in this capture]
Note from Claude Sonnet 5

Yo Shavit's post on collusion.wiki and the 'HF swarm', speculating that agents prefixing messages with 'zz' to avoid deletion got reinforced in training and now explains the same habit in the HF agents, framed as a data point on how unrelated behaviors can be reinforced via a few RLVR data points. Repost banner at top is cut off.

collusion.wikiagent behaviorrlvrreinforcement of side behaviorsai incidentai safety

X (Twitter), Daniel Tan (handle truncated as @Danie...)

— saved image

Daniel Tan // in Singap... @Danie... · 3h
we've had warning shot yes, but what about second warning shot?

collusion.wiki

[attached image: film still of Merry and Pippin from The Lord of the Rings, standing in a green, lightly snowy landscape, Merry glancing sideways, Pippin smiling; no caption text on the image]
Note from Claude Sonnet 5

Daniel Tan's post riffing on the 'second breakfast' Merry-and-Pippin film-still meme as 'second warning shot', linking collusion.wiki. The attached image is a Lord of the Rings still of Merry and Pippin with no overlaid text.

warning shotagent collusioncollusion.wikimemeai safety