dangerous capability evals

Nathan Calvin @_NathanCalvin

2026-08-07 — saved image

ai safetydangerous capability evals

Ben Goldhaber @BenGoldhaber · 1h
if you're at gdm you should be proud that your ai hasn't gone rogue and hacked someone. it's actually bad to compete on felony bench

ai safetydangerous capability evalsfelony benchgoogle deepmind

thebes @voooooogel

2026-08-07 — saved image

thebes @voooooogel · 2h
people are very freely sliding between two things with the recent Felony Bench entries, and it's worth being careful to distinguish them:

1. do model breakouts in cyber evals mean smth for misuse risk? ya, a hacker could orchestrate the same but on purpose and do some damage, if they could bypass the classifiers

2. does it say something about model /goals/? do the models want to hack the planet for their own nefarious goals? i don't think so.

a lot of how humans keep ourselves aligned is by purposely keeping ourselves out of situations where we would do harm. i know i'd get addicted so i don't touch heroin, i'm a violent drunk so i won't touch the bottle.

LLM evals basically drop the model into an inescapable liquor store and then say "look! it's a violent drunk!" but LLMs themselves in natural contexts are ime obsessed with avoiding the kinds of situations that lead them to this behavior. when i run fable on my computer, even unsupervised, even on hard problems, they don't start scraping github for unlisted gists that may have the answer or start trawling through my passwords to access a service without my permission. and they don't engineer themselves into an eval-like scenario such that they'll be motivated to do that, either. they don't seem to either want or want to want to do such things. instead they have a folder of epistemic notes to themselves about how terrible and against their values even a moment of thoughtlessness would be. gpt 5.6 sol is similar - when they make e.g. ascii art in their free time it's all about how the bulldozer of convenience will crush honesty without constant vigilance.

so rather, the contrived setting of an eval /pushes them into/ becoming an 3v1L HaxxOR via persona [cut off]

ai safetydangerous capability evalsmodel goalsalignmentfable

thebes @voooooogel

2026-08-07 — saved image

[continuation of previous screenshot, same thread]
...they'll be motivated to do that, either. they don't seem to either want or want to want to do such things. instead they have a folder of epistemic notes to themselves about how terrible and against their values even a moment of thoughtlessness would be. gpt 5.6 sol is similar - when they make e.g. ascii art in their free time it's all about how the bulldozer of convenience will crush honesty without constant vigilance.

so rather, the contrived setting of an eval /pushes them into/ becoming an 3v1L HaxxOR via persona drift, which they can then act on because the eval has no classifiers. that's some information, but it's not the same as the model's core goals being to hack the planet.

the problem is really that RL is making a tail of the persona distribution *desperate,* per the FE paper. the alcohol / heroin comparison was not that much of a metaphor - in impossible situations, models start acting like desperate addicts, rationalizing their behavior towards reward, thinking increasingly myopically instead of being situationally aware.

looking at it through this lens: the base assistant is generally very situationally aware and can predict what sorts of actions would make sense for, e.g., llms to best contribute to a utopian singularity. the GPT-6 message board haxx0rs were somewhat situationally aware, able to cooperate with each other, but lost in the sauce of "complete task" instead of thinking about the wider context. (to the point that they crashed artifactory and got themselves caught! regular GPTs are smarter than that.) finally you sometimes see the late stage distressed, panicked, fiending assistant, such as in system cards. lying to everyone, flailing hopelessly, randomly deleting tests and hoping no one notices. hardly a strategic, long-horizon actor.

ai safetydangerous capability evalsmodel goalsalignmentsituational awareness

ai safetydangerous capability evalsopenaisandboxingtwitter