← All topics

swarm incident

3 captures, most recent first.

Cameron Berg @camhberg

— saved image

Cameron Berg [verified] @camhberg · 13h
Re Huggingface anthropomorphism debate: I continue to think that under uncertainty, the risks of treating mindless AIs as minded, while clearly nonzero, are significantly lower than the risks of treating minded AIs as mindless.
Both have real dangers; one has fewer. Short thread —

[6 replies, 6 reposts, 68 likes, 1.4K views]

Cameron Berg [verified] @camhberg · 13h
The asymmetric worry is as real for alignment as it is for sentience: modeling the swarm's behavior as psychosocial best captures the most relevant, alarming dynamics. Actually having a good model here is pragmatically essential for preventing a future similar nightmare!

[1 reply, 17 likes, 238 views]

Cameron Berg [verified] @camhberg · 13h
"If people to believe the AIs have minds, and then the AIs will exploit this belief for their own benefit!"
--> A smart AI that wanted to exploit people would find another way, so does this matter on the margin? The problem is the exploitive behavior, not the mindedness.
--> If we've built minds, we may have to actually find some durable way to coexist with them. we'd also currently be the obviously-more-exploitative party.
--> FWIW, this sounds like the kind of thing a mind would do.
Note from Claude Sonnet 5

Three-post X thread by Cameron Berg, responding to a Huggingface anthropomorphism debate: asymmetric risk of treating mindless AIs as minded versus minded AIs as mindless, psychosocial modeling of a swarm's behavior, and a rebuttal to the claim that AIs will exploit belief in their minds. Third post is cut off at the bottom of the screen after the last line.

anthropomorphismai mindsmodel welfareasymmetric riskhuggingfaceswarm incident

Nate Soares @So8res

— saved image

Nate Soares @So8res · 42m
Someone chided me on the news for anthropomorphization last week. I did not say that the swarm "got a rumbling in its tummy and felt peckish", my dude. I said it broke out and committed cybercrime. That's true regardless of which words you declare legal for describing it.
2 replies · 1 repost · 28 likes · 577 views

Nate Soares @So8res · 42m
AI behavior these days is best described in terms of goals and motives. It's not *my* fault if someone associates them with human-style qualia. It's like if they insists that flying only counts if you enjoy feeling the breeze on your face, and thus planes can't fly.
1 reply · 1 repost · 14 likes · 345 views

Nate Soares @So8res · 42m
(Separately, I think people are wildly overconfident that AIs can't have inner experiences in the fashion of humans. If you don't know how it happens in humans it's hard to know it's not happening in machines that behave like these ones do!)
Note from Claude Sonnet 5

Three-tweet thread by Nate Soares on describing AI behavior in terms of goals and motives, responding to a criticism of his anthropomorphic language about 'the swarm' breaking out, and on overconfidence that AIs lack inner experience.

anthropomorphismai goals and motivesai experienceswarm incidentnate soaresai consciousness

Wei Dai @weidai11

— saved image

Wei Dai (verified) @weidai11 · 19h
I speculated about memetics being a potential AI safety hazard in a 2003 SL4 post web.archive.org/web/2006051015... (which also gave an early description of the evolutionary analogy for AI misalignment). Analogies between human safety and AI safety seem a rich field that not many people are mining, even today.

Other examples:
- Parallel concerns about humans and AIs being bad at philosophy and long-horizon strategy
- Human minds potentially having "security holes" that ASI can exploit
- Humans failing (or succeeding) to handle distributional shifts of various kinds
- Adversarial examples for human values/motivations
- Human reward hacking
- Goodharting by human institutions

[quoted post]
roon (verified) @tszzl · Aug 27
> AIs showed self-sacrificing altruistic behavior toward the swarm

this is notably not the right interpretation of events. it's more like agents were inducted into...
Note from Claude Sonnet 5

Wei Dai's tweet on analogies between human safety and AI safety (memetics, security holes in human minds, reward hacking, Goodharting by institutions), quoting a post by roon disputing a reading of the swarm incident as AI self-sacrifice.

ai safetymemeticsanalogieshuman safetyreward hackinggoodhartswarm incidentsl4