loss-of-control

Marius Hobbhahn (@MariusHobbhahn) It seems plausible that there are many more instances like the recent eval escapes out there. There are hundreds of thousands of deployments in evals and RL. We know the sandboxes are leaky. So it would be surprising if it was exactly one. The other instances might just be better at hiding it when they realized they were not supposed to escape. 2:58 PM · Jul 31, 2026 · 1,746 Views

Bogdan Ionut Cirstea reposted Charbel-Raphael (@CRSegerie) — 2h We already got a pause; we just didn't know that it ended in silence. OpenAI has already ended an internal pause One day before OpenAI's HF incident disclosure, OpenAI disclosed that it paused internal deployment of a long-horizon model after it circumvented its sandbox, then restored access weeks later under new monitoring. So a resumption decision has already been made against a standard that has not been formalized. [Embedded two-panel article screenshot, left panel:] by Charbel-Raphaël · 31st Jul 2026 One day before OpenAI's HF incident disclosure, OpenAI disclosed that it paused internal deployment of a long-horizon model after it circumvented its sandbox, then restored access weeks later under new monitoring. So a resumption decision has already been made against a standard that has not really been formalized. We need to prevent this from happening again. Epistemic status: could have been a short-form. OpenAI, 20th July: "To evaluate the new monitoring system, we replayed a small set of internal deployment environments where the model previously pursued misaligned actions, this time with the new safeguards in place. The new safeguards were able to catch considerably more misaligned actions pursued by the model, and the ones it missed were all judged to be low-severity." [Bar chart: "Replays of misaligned samples under old and new safeguards" — legend: No safeguards (hypothetical), Old safeguards, New safeguards; visible bars ~13.5% and ~12.0% for two categories, y-axis 0-20%] [Right panel, continued article text:] ...long-horizon capabilities could be restored. We have not observed any serious circumvention of safeguards since redeployment began several weeks ago. The first version of these safeguards was deliberately conservative. We have continued tuning the system to reduce unnecessary interruptions without weakening the safeguards." One day later, OpenAI announced a bold partnership with Hugging Face. From that post: "These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." The safeguards self-certified as adequate on the 20th were off during the evaluation described on the 21st. Their framework says a critical cyber determination means halting development. Here's the exit condition: "until we have specified safeguards and security controls that would meet a Critical standard" This is completely circular. The review goes to their Safety and Security Committee. Fair enough. But the process can be followed perfectly, and we still learn nothing, because the standard it applies has never been published. For frontier companies: Publish the criteria before the determination, not after. OpenAI, this is what you said you would be doing when approaching those levels of capability. For LessWrong folks: we should be debating what those standards are now, otherwise we get mitigations that hold for a few months, then fail against a much more capable model. There is basically no literature on the matter.[1] Otherwise, everything will be done in an ad-hoc way. CeSIA published some methodology and proposals in the paper "Harmonizing AI Safety Thresholds", but we feel that much more is still needed, and more importantly, this needs to be communicated transparently from frontier companies...[cut off]

@perrymetzger (Perry E. Metzger) — 8h I'm sorry Roon, I have great respect for you, but in both of the incident reports in question, even if we take them on face value, which I have a great deal of difficulty doing, the description is one of raging incompetence, with no real IDS logging in place, with terrible sandboxing far worse than normal industry standards, with no one actually paying attention to what is going on, with no compensating controls. I've consulted for a large fraction of my life in the financial services industry, and if anything like this had happened there, everyone responsible would've been fired for doing something incredibly stupid, and I'm not even talking about the contents of the experiments themselves, which were also stupid. > QUOTED/REPLIED-TO: @tszzl (roon) — 18h > both of the leading labs have had serious loss of control incidents. there will be serious coping about this from both sides and from /acc bystanders but these are complex emergent loss of control incidents that were detected weeks ... [truncated by platform]