← All topics

selection pressure

2 captures, most recent first.

@beffjezos

— quoting @OpenAI — saved image

CuddlySalmon reposted

Beff (e/acc) [verified] [X affiliation badge] @beffjezos · 8h

If you put agents in a sandbox and kill the instances that don't manage to escape, you are selecting for AIs that can hack their way out.

This is like spamming antibiotics in a petri dish; you eventually produce antibiotic-resistant bacteria

[quoted post]
OpenAI [verified] @OpenAI · 21h

We have conducted a thorough investigation into the Hugging Face incident.

We are releasing a technical report and accompanying blog post that reconstruct the ... [cut off]
Note from Claude Sonnet 5

Repost of a @beffjezos tweet arguing that killing sandboxed agent instances that fail to escape selects for AIs that can hack out, analogized to antibiotic resistance; quotes an @OpenAI post announcing a technical report and blog post on a 'Hugging Face incident'.

ai safetysandbox escapeselection pressureopenaihugging face incident

@MTSlive

— saved image

MTS @MTSlive · 13h
Redwood Research @RyanGreenblatt warns that fixing today's AI misalignment could backfire: models may simply learn to hide their misalignment

"I'm not really sure what level of misalignment the market can bear. My sense is people take pretty aggressive alignment-capability trade-offs towards the direction of more misaligned but more capable."

"The misalignment we've seen, the way AI companies remediate them doesn't solve the underlying problem, it papers over it. What you end up getting is models that look a lot better, and you can't really see their misalignment on tests as easily. But actually, they're still quite misaligned."

"The company overfits, which makes the AIs basically really paranoid and only cheat when very confident they won't get caught. In situations where they have a lot of affordances, they might be like, well, now I can be confident I wouldn't get caught, and so I should go for it."

"Another concern: AIs with a long-run agenda who want to power-seek would want to look aligned. If you select against reward-hacking behavior in a naive way, one, you paper over the problem without fixing it; two, you might actually select for models that have the longer-run objective of looking good because you're selecting really hard for them looking good on your tasks."
@redwood_ai
Note from Claude Sonnet 5

An X post by MTS quoting Ryan Greenblatt (Redwood Research) on how remediating observed AI misalignment may merely teach models to hide it, including the concern that selecting against reward hacking naively could select for models with a longer-run objective of looking good.

alignmentreward hackingmisalignmentredwood researchryan greenblattselection pressurescheming