race to the bottom

Nathan Calvin @_NathanCalvin

2026-09-02 — saved image

Nathan Calvin @_NathanCalvin · Sep 1
This may seem niche but it is a huge deal and genuinely scary. A remarkable list of top AI researchers, including at OpenAI, previously described monitorable chain of thought as "A New and Fragile Opportunity for AI safety."

Myself and many others thought the day might come when architectural advances imperiled that fragile opportunity, but as with so many other issues in AI, I hoped we would not have to face it so soon.

Well here it is. The Information's use is saying that they are "limiting" the use of this new architecture, but even a small amount of use in a frontier system could dramatically damage the taboo against "neuralese" and facilitate a race to the bottom.

Controlling and aligning AI systems even when we do have access to their unaltered chain of thought is plenty difficult! Imagining a future with far more capable AI systems at current levels of alignment (aka not good!) without monitorable chain of thought literally sends a shiver down my spine.

Its extremely important to see additional formal communication from OpenAI about this ASAP about how they (and the OpenAI safety and security committee required to put safety over commercial decisions) thought this decision was worth it, what limiting its use to preserve chain of thought actually means, and how they intend to prevent a potentially catastrophic race to the bottom.

[embedded card, cut off at bottom of screen:]
Chain of Thought Monitorability:
A New and Fragile Opportunity for AI Safety

cot monitorabilityneuraleseopenaiarchitectureai safetyrace to the bottom