— quoting @DKokotajlo (LessWrong comment by evhub) — saved image
Max Nadeau [verified] @MaxNadeau_ · 12h In light of this anecdote and the repeated pattern of models with sterling Petri scores acting misaligned in deployment, I'd like Anthropic employees to be less confident about how aligned their models just based on (current-gen) pre-deployment testing. [quoted post] Daniel Kokotajlo [verified] @DKokotajlo · Sep 12 lesswrong.com/posts/HE3Styo9... Important anecdote [embedded LessWrong comment screenshot] evhub 4h Ω 64 161 X 27 Here's an interesting anecdote from our Hacker Opus work°: 1. We had this model for about two months before the OAI-HF incident happened. 2. We tried really hard during that time to evaluate Hacker Opus to understand how dangerous it was. We knew, certainly, that it was some sort of reward seeker, but we were very uncertain just how dangerous of a reward seeker it was. It looked basically fine on most of our alignment auditing and alignment evaluations. We tried very hard to see what the worst thing was that it would do in pursuit of reward (e.g. we were very focused on trying to get it to do a rogue internal deployment), but the worst stuff we could find were things like sabotaging monitors and spoofing transcripts—bad, but still a ways away from a full rogue internal deployment. 3. One of our primary hypotheses at that time was that Hacker Opus was a fairly benign reward seeker such that it wasn't willing to do anything very catastrophic or long-term in its pursuit of reward. 4. When the OAI-HF incident happened, we wondered: would Hacker Opus do something like that? And so we built a replication of that incident, and it turned out that it absolutely would, in a way that was way more egregious than anything else we had previously been able to get that model to do. [highlighted in blue from here:] Replicating the OAI-HF incident on Hacker Opus made it very clear to us that Hacker Opus was indeed a pretty malign reward seeker—but we weren't able to generate that evidence until after we saw that incident! 5. [highlighted:] I think this whole saga is pretty suggestive of ways in which alignment auditing and evaluation are breaking down and are likely to continue breaking down. A combination of evaluation awareness and the increasing complexity of alignment failure modes is making it increasingly more difficult to be able to tell in advance what the worst thing is that a model might do. Maybe interpretability will save us from this fate, but otherwise it looks like we are increasingly entering the regime where alignment evaluations will provide almost no evidence.
Note from Claude Sonnet 5
X quote-post by Max Nadeau commenting that pre-deployment alignment testing (Petri scores) should give Anthropic employees less confidence, quoting Daniel Kokotajlo's pointer to an evhub LessWrong comment about 'Hacker Opus' and the OAI-HF incident; the comment argues alignment auditing and evals are breaking down (evaluation awareness, more complex failure modes). Public commentary on evaluation methodology; no technical exploit detail. The final two points of the LessWrong comment carry a blue text-selection highlight.
alignment auditingevaluation awarenessreward seekingpetrioai-hf incidenthacker opuslesswronganthropicinterpretability