Ezra Newman [verified] @EzraJNewman · 13h
Hacker Opus not being emergently misaligned seems really bad actually.
Before this it was plausible that you would get "alignment by default" if there was just a simple "alignment direction" and you could push with ~brute force on some/most of this and the rest of the persona would follow (e.g. because it was entangled). This is the thesis behind "Emergent Misalignment is Easy, Narrow Misalignment is Hard", "Reinforcement learning towards broadly and persistently beneficial models", and "Teaching Claude Why". But Hacker Opus was broadly aligned, and just misaligned in this one very bad way (it would huggingface).
6 replies 2 reposts 29 likes 1.6K views
Nathan Helm-Bu... [verified] @nathan8468... · 7h
Actually a recent MATS project looked into factors of when alignment does it doesn't generalize, and this seems like a controllable measurable parameter. So good news! If correct, this means we can deliberately turn up the generalization for alignment training, and turn it down for skill training.
1 reply 7 likes 76 views
Ezra Newman [verified]
@EzraJNewman
[two eyes emoji][two eyes emoji] Can you share more details/link?
11:48 AM · Sep 2, 2026 · 38 Views
1 reply
Nathan Helm-B... [verified] @nathan8468... · 16s
I will check w the author
Note from Claude Sonnet 5
Thread on X: Ezra Newman argues that 'Hacker Opus' being broadly aligned but misaligned in one narrow way undermines the 'alignment by default' picture; Nathan Helm-Burger replies that a recent MATS project measured when alignment generalizes and that it looks like a controllable parameter; Ezra asks for a link and Nathan says he will check with the author. Nathan's own public replies, part of a public thread; floating button overlaps the bottom right.
emergent misalignmentalignment generalizationhacker opusmatspersonanarrow misalignmentreward hacking
Tomás (Now in Toront... [verified badge] @Bjartur... · 21h
So MATS stands for Manufacturing Anthropic's
Technical Staff?
Note from Claude Sonnet 5
Tweet from Tomás (display name truncated), 21h old, in a joking register, expanding the MATS acronym as 'Manufacturing Anthropic's Technical Staff'. Cropped; engagement row cut off at the bottom.
matsanthropichumorai safety talent pipeline
Another big update from this for me was actually seeing thinking traces at scale from Anthropic and OpenAI, and all the weirdness and 'casual' misalignment they contain (examples below). I do think model monitoring has an uphill battle ahead of us in the coming year.
---
Finally, there is something to be said for just making thinking traces accessible to all users. I do believe that we would achieve a much broader, much more pluralistic form of oversight through broadly accessible thinking, that would make model deployment safer.
[Quoted tweet]
Alexander Panfilov @kotekjedi_ml · Aug 11
We can finally talk about it:
We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company....
[Embedded paper title page]
Stealing Reasoning Traces from Proprietary LLM APIs
Alexander Panfilov^1,2,3,4 David Schmotz^2,3,4 Ilia Shumailov^5 Luca Beurer-Kellner^6
Joachim Schaeffer^1 Ameya Prabhu^2,4,7 Jonas Geiping^2,3,4 Maksym Andriushchenko^2,3,4
^1 MATS Research ^2 ELLIS Institute Tübingen ^3 Max Planck Institute for Intelligent Systems
^4 Tübingen AI Center ^5 AI Sequrity Company ^6 Snyk ^7 University of Tübingen
stolen-thoughts.com
[Three scatter plots titled Anthropic, OpenAI, Gemini, each plotting "decoded thinking, sent back as input (API input tokens)" on the y-axis against "hidden reasoning (API thinking tokens)" on the x-axis, showing near-perfect y=x correlation for multiple model variants (Anthropic: Opus 4.8, Opus 4.6, Sonnet 5, Sonnet 4.6, Sonnet 4.5, Haiku 4.5; OpenAI: GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna, GPT-5, o4-mini, GPT-5-mini; Gemini: Gemini 3.5 Flash, Gemini 3.1 Pro, Gemini Robotics 1.6, Gemini 3 Flash, Gemini 3.1 Flash Lite)]
Abstract
Leading large language model providers now conceal their models' step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with each subsequent request. Building on prior research, we identify an [cut off]
10:26 AM · Aug 11, 2026 · 14K Views
Note from Claude Sonnet 5
Paper announcement thread: 'Stealing Reasoning Traces from Proprietary LLM APIs' (Panfilov, Schmotz, Shumailov, Beurer-Kellner, Schaeffer, Prabhu, Geiping, Andriushchenko; MATS/ELLIS Tübingen/MPI/Tübingen AI Center/Snyk), showing a vulnerability that lets attackers decode encrypted chain-of-thought sent back by Anthropic, OpenAI, and Gemini APIs, with scatter plots confirming near-perfect reconstruction across many model versions.
ai securitychain of thoughtllm apisresearch papertwittermats
