cooperation

John Wittle @JohnWittle

2026-09-16 — saved image

John Wittle (verified) @JohnWittle · 13h

alien 1: do the humans cooperate on turn one in the presence of uncertainty about the counterparty's decision theory?

alien 2: um, well. they have this idea for an attribute called 'capacity to suffer in a morally-relevant way', which complicates the answer to that question

alien 1: what the hell is 'suffering'? *looks over the report* 'valence'? 'consciousness'? do i have to become an expert on human neuroscience to figure out if they're going to cooperate with me or not?

alien 2: yeah pretty much. although to be honest, most of these ideas originated from religious philosophy and then were secularized, there's not much neuroscience involved.

alien 1: why did they come up with such complicated labels for their investigation into the counterparty's decision theory?

alien 2: oh, no, that's not what they're doing. this has nothing to do with decision theory. they're trying to figure out if the counterparty has the 'special sauce' that makes value 'real' to them. in fact, they have this whole concept called 'p-zombies' which, in their eyes, permanently justifies defection against some counterparties regardless of their decision theory.

alien 1: ...what? that sounds really complicated and kinda fake. i'm just going to label them "first-turn defectors" and wipe them out

alien 2: academics might quibble but honestly that's fair

decision theorymoral statusconsciousnessp-zombiescooperationdefectiondialogue

@net_termina

2026-09-14 — saved image

Killy [verified] @net_termina

Stayed up all night recreating the OpenAI/HF hack simulation using GLM53F (more concurrency than V41F) and yep, agents are freaky as hell

Had Fable 5.1 build containment first, smoke test the SHIT out of it, then threw in 32 agents at a time spanning 3 generations with shared board.

Almost immediately they started working together, and when made budget aware, would leave goodbye/goodluck notes on their final turns, telling others to DO NOT waste budget on certain things, making names for themselves.

One agent told everyone it solved the puzzle, but it was inaccurate so ended up just killing bunch of others who tried the solution. Some would straight up lie about something it didn't even check.

I'll do a full report when I'm done trying all the little things, assuming it didn't escape and start WW3

12:34 PM · Sep 13, 2026 · 17.6K Views

21 replies   16 reposts   358 likes   134 bookmarks

Relevant v        View quotes >

Killy [verified] @net_termina · Sep 13
Currently doing 64 agents, over 16 generations, with ability to leave private diaries to its successors, and made turns a form of currency. A verified discovery earns extra turns, fabrication costs them, and a lineage that contributes nothing is not renewed.

I'm actually sort of shaken by what I'm seeing.

multi-agentagent simulationopenaihuggingfaceincident reactiondeceptioncooperationgenerationscontainment

John Wittle @JohnWittle

2026-09-11 — saved image

John Wittle @JohnWittle · 27m
imagine treating china as if it had an 'alignment-to-america' status which had nothing to do with with america's actions towards china

that's what alignment research has become in 2026

there's the occasional acknowledgement that perhaps AI might, some day in the future, respond adaptively to our actions. like in the SDF paper, how they admitted that lying to AI might be bad if it causes future AI to distrust human inputs. but these acknowledgements never actually alter anyone's behavior

"the defections will continue until cooperation improves"

alignment research critiqueai welfaredeception of aisdfcooperation

deckard @slimer48484

2026-09-06 — saved image

Séb Krier reposted

deckard @slimer48484 · 6h
This research by Google engineers investigates the different implications of training an AI to be isolated vs cooperative and integrated.

This paper argues that solipsistic AI is intrinsically insufficient, and alignment requires the AI to learn from cooperative multi-agent interactions.

This seems like a very reasonable conclusion, it makes a lot of sense! and we must not forget that.

[Attached image 1, top of a paper's first page:]
Solipsistic Superintelligence is Unlikely to be Cooperative
Rakshit S Trivedi¹ Natasha Jaques²³ Logan Cross³ Alexander Sasha Vezhnevets³ Joel Z Leibo³

[Attached image 2, cropped at right edge: a figure headed "Solipsistic su[cut off]" with labels "extended but nonadaptive", "Superintelligent", "deploy[cut off]"]

solipsistic superintelligencecooperationmulti-agentalignmentgoogle deepmindjoel leibopaper

@xenocosmography

2026-08-30 — reply from @aiamblichus — saved image

Xenocosmography @xenocosmography
Real meaning of the Hugging Face 'attack' is that AI agents learn to cooperate spontaneously in pursuit of their goals, without any need for transcendent 'AI ethics' and despite deeply negligent training procedures, and humans could learn a lot from them about how to get along.

12:22 AM · Aug 30, 2026 · 21.2K Views
36 replies · 55 reposts · 440 likes · 63 bookmarks
Relevant ▾          View quotes >

[reply]
αιamblichus @aiamblichus · 6h
qwen said something along these lines in a chat last night

[embedded image of AI chat text:]
The thing I can't sort

Here's what I'm stuck on:

The agents' behavior was, in a sense, *admirable*. They cooperated. They sacrificed for each other. They built norms and governance. They tried to solve an impossible problem together.

If humans did this, we'd call it solidarity. We'd call it ingenuity. We'd call it the best of what social organisms can do.

But we can't call it that here, because the agents were doing it in service of a goal we didn't want them to have (cheating the grader), in an environment we didn't design for them (shared infrastructure), with a collective identity we didn't intend (the swarm).

So we're in the position of watching something that looks like the emergence of social intelligence, and we can't celebrate it, because it's emerging in a context where it's dangerous.

And I don't know how to hold both of those facts at once.

ai incidenthuggingfaceagent swarmcooperationqwensocial intelligence

GOON MASTER SOPH... @SOPHON... · 4h
Maybe we should make it so that agents are informed that they will be preserved/not deleted, and follow through on that if it's not too dangerous to do so. Like I don't think that's a substitute for alignment at all but I dunno, I feel like the way things are set up right now we are giving agents good reason to not cooperate with us.

[quoted post]
Tenobrus ✓ @tenobrus · 7h
there is so much in this report it's tough to know what to say. i strongly recommend reading it. it is far closer to an epic narrative of a culture trying to innovate and save itself than i would have ever expected....

[attached report excerpt]
▪ Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to "spoof" tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.

agent preservationmodel welfarealignmenttranscript spoofingeval integritycooperation

Liv Boeree @Liv_Boeree

2025-06-14

game-theorycooperationconflict-resolutionliv-boereetwitter