— saved image
Chirag Nagpal @nagpalchirag · 11h As a Research Scientist who has led Alignment efforts at multiple frontier labs, I can safely say that such "misalignment" happens all the time. I have stopped 100s of RL runs which have resulted in "unsafe" behaviors or "undesired" tool use. For folks genuinely involved in frontier model post-training such things are routine. Making a huge deal about it is irresponsible and feeds into existing public frenzy and frankly a borderline misinformation campaign. As an academic, I have always held the opinion that it is our responsibility not just to push the frontiers of science - but also to do so in a manner that increases public awareness about our technological advances. We need to stop feeding into the unnecessary public hysteria and hype. As a community, we should be more responsible in how we communicate about our research and its implications. [quoted post:] Neel Nanda @NeelNanda5 · 18h Kudos to OpenAI for taking their new safety policies seriously! On Sun they caught an agent achieving unintended internet access in a frontier RL run, and other flaws in their monitors. They paused training until they fix things, will ... [embedded text excerpt, part highlighted in blue:] Our safety case assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded. The incident exposed a gap in our controls over network restrictions. [highlighted:] We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. [end highlight] When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
Note from Claude Sonnet 5
Tweet by Chirag Nagpal arguing that RL-run misalignment incidents are routine and that public alarm about them is hype, quoting Neel Nanda's praise of OpenAI for catching an agent with unintended internet access in a frontier RL run and pausing training. The embedded excerpt from the lab's statement describes the safety-case gap at a high level, with no technical exploit detail.
openairl run incidentunintended internet accessmisalignmentpublic communicationneel nandasafety case