— quoting @amir — saved image
Thomas Larsen @thlarsen Very bad if true. I previously thought that in a short timelines world, the most likely case was that (1) the AIs would be misaligned, but (2) we would get a lot of evidence about it from reading the COTs. This evidence would increase the chance of a reasonable response from labs/governments. Now I still think the AIs are going to be misaligned, but that we won't have the ability to tell (we have to rely on toolcalls/agentic behaviour instead of the COT) and even if we do, the investigation will be nearly impossible because we'll have to trust the AI to self report what it was thinking about. This is also an example of things going faster than AI 2027 -- we had Neuralese starting in March 2027 [embedded image of AI 2027 text:] March 2027: Algorithmic Breakthroughs Three huge datacenters full of Agent-2 copies work day and night, churning out synthetic training data. Another two are used to update the weights. Agent-2 is getting smarter every day. With the help of thousands of Agent-2 automated researchers, OpenBrain is making major algorithmic advances. [highlighted: One such breakthrough is augmenting the AI's text-based scratchpad (chain of thought) with a higher-bandwidth thought process (neuralese recurrence and memory).] Another is a more scalable and efficien[cut off] way to learn from the results of high-effort task solutions (iterated distillation and amplification). [quoted post:] Amir Efrati @amir · Sep 1 new: OpenAI & others quietly using loop transformers that don't show their 'thinking' when scaled up a leap forward on performance, but sparking concerns inside & outside OpenAI re: security as this takes off [embedded The Information article card, cut off at bottom of screen]
Note from Claude Sonnet 5
Thomas Larsen's X post reacting to a reported shift to looped transformers whose reasoning is not visible, arguing it removes CoT evidence of misalignment and noting AI 2027's neuralese milestone; quotes Amir Efrati's post. Screenshot is cut off at the bottom by the app's navigation bar.
neuraleselooped transformerscot monitorabilityai 2027openaimisalignmentthe information