neel nanda

roon @tszzl

2026-10-05 — reply thread: @tszzl, @NeelNanda5, @tablefourthree — saved image

roon @tszzl · 23h
this is the same model from a very long time ago now
3 replies · 1 repost · 109 likes · 5.5K views

Neel Nanda @NeelNanda5 · 23h
Fine, how do I keep learning that openai internal models are even scarier and more concerning than I thought they were?
4 replies · 1 repost · 93 likes · 2.6K views

chair @tablefourthree · 22h
This report doesn't seem that worse than Anthropic's famous agent blackmail test. So it doesn't seem like new risky behavior the public didn't know about in general. (And I imagine you and every lab would have seen stuff like this in evals already?)
1 reply · 2 likes · 465 views

Neel Nanda @NeelNanda5 · 21h
The blackmail thing was contrived BS with a prompt carefully designed to elicit bad behavior by human red-teamers, this seems real and unintentional

openaimisalignment disclosureagentic misalignmentblackmail evalneel nandared-teaming

@nagpalchirag

2026-09-27 — saved image

Chirag Nagpal @nagpalchirag · 11h
As a Research Scientist who has led Alignment efforts at multiple frontier labs, I can safely say that such "misalignment" happens all the time. I have stopped 100s of RL runs which have resulted in "unsafe" behaviors or "undesired" tool use.

For folks genuinely involved in frontier model post-training such things are routine. Making a huge deal about it is irresponsible and feeds into existing public frenzy and frankly a borderline misinformation campaign.

As an academic, I have always held the opinion that it is our responsibility not just to push the frontiers of science - but also to do so in a manner that increases public awareness about our technological advances.

We need to stop feeding into the unnecessary public hysteria and hype. As a community, we should be more responsible in how we communicate about our research and its implications.

[quoted post:]
Neel Nanda @NeelNanda5 · 18h
Kudos to OpenAI for taking their new safety policies seriously! On Sun they caught an agent achieving unintended internet access in a frontier RL run, and other flaws in their monitors. They paused training until they fix things, will ...

[embedded text excerpt, part highlighted in blue:]
Our safety case assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded. The incident exposed a gap in our controls over network restrictions. [highlighted:] We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. [end highlight] When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.

openairl run incidentunintended internet accessmisalignmentpublic communicationneel nandasafety case

Neel Nanda @NeelNanda5

2026-08-07 — quote-tweeting @Singh_Aditya1 (Aditya Singh) — saved image

Neel Nanda [verified] @NeelNanda5 · 15m
If we are to understand scheming models, we need to understand a model's motivations. Current models don't seem to want to take over the world, but they LOVE to cheat on tests (as HuggingFace experienced...)

We do a deep dive into what's going on psychologically for models here

Aditya Singh [verified] @Singh_Aditya1 · 21h
Recent events show we need a mature science of "Model Forensics" to investigate concerning behavior. But how do we study misalignment? Current models aren't paperclip maximizers, but they do cheat and misrepresent their work ...

[Embedded infographic, dark blue background]
MODEL FORENSICS
Why do models task game?
1. Sensitive to oversight — Models cheat less when they expect their work to be checked.
2. Beyond instruction following — Models ignore explicit instructions to revert their failed work.
3. Delusion or deception — Some models fool themselves; others knowingly deceive.
4. Misleading final outputs — Models fabricate results — with no hint of it in their chain of thought.
5. A "bullshitting" propensity? — Models that make up answers in chat also cheat on real tasks.

ai safetyschemingmisalignmentmodel forensicshugging facetwitterneel nanda

Neel Nanda @NeelNanda5

2025-03-23

twitter/xneel nandaacademiacitationsinterpretability research community