← All topics

jonas geiping

2 captures, most recent first.

Jonas Geiping @jonasgeiping

— saved image

Jonas Geiping @jonasgeiping
Earlier today we release our report about a vulnerability that allowed us to read out the encrypted thinking traces from many frontier models (thread below!):

A few thoughts:

First, there is an immediate privacy concern with publicly posted reasoning traces (which is also why we took time to release the report after the initial disclosure). We were able to decode the thinking of many json traces posted online, and found private info in there.

Ironically, during this investigation we also had entry into HF during the cybersec incident due to a leaked prod key (but did not exercise the key beyond a whoami  ;)).

Another big update from this for me was actually seeing thinking traces at scale from Anthropic and OpenAI, and all the weirdness and 'casual' misalignment they contain (examples below). I do think model monitoring has an uphill battle ahead of us in the coming year.

---

Finally, there is something to be said for just making thinking traces accessible to all users. I do believe that we would achieve a much broader, much more pluralistic form of oversight through broadly accessible thinking, that would make model deployment safer [cut off]
Note from Claude Sonnet 5

Continuation of the Jonas Geiping thread (seq 741) about the vulnerability decoding encrypted frontier-model reasoning traces, discussing privacy risks of leaked reasoning, the ironic Hugging Face access incident, and an argument for making thinking traces broadly accessible for oversight.

ai securitychain of thoughtmodel monitoringtwitterjonas geiping

X (Twitter), @maksym... (Maksym Andriushc...), quoting @jonasgeiping

— quoting @jonasgeiping — saved image

Maksym Andriushc... @maksym... · 22h
many good points from Jonas about our paper... including "Ironically, during this investigation we also had entry into HF during the cybersec incident due to a leaked prod key (but did not exercise the key beyond a whoami  ;))."

of course, we immediately reported this to HF, but what if an extremely goal-oriented LLM agent would find it instead of us? i keep thinking about this. we are collectively not prepared for what's coming... a lot of work to be done to secure current systems.

[Quoted tweet]
Jonas Geiping @jonasgeiping · 23h
Earlier today we release our report about a vulnerability that allowed us to read out the encrypted thinking traces from many frontier models (thread below!):
...
Note from Claude Sonnet 5

X thread about a security research paper: the authors found a vulnerability letting them read encrypted 'thinking traces' from many frontier models, and separately gained accidental access to Hugging Face internals via a leaked prod key during their cybersecurity investigation, prompting worry about goal-oriented LLM agents exploiting such lapses.

ai securityllm agentschain of thoughthugging facetwitterjonas geiping