Geoffrey Irving

Geoffrey Irving @geoffreyirving · 16h
Dan with the key point! It is not remotely enough that we'll be defensive-dominant in the limit: the entire path has to work out, and the path will be very rough. We'll be in "exponentially harder is not hard enough" territory for a while yet.

[quoted link card]
naml.us
Exponentially harder isn't hard enough yet
In well designed cryptographic security systems, the attacker needs to do exponentially more work than the defender in order to read a secret, forge a message, etc., subject to appropriate...
Geoffrey Irving · Tuesday, 3 July 2012

[quoted tweet]
Dan Lahav @dan_lahav · 19h
[link card image]
The End-State Fallacy: Where Is AI Security Going?
Frontier AI models had a giant performance gain in coding in the ...

ai securitycybersecuritycryptographytwitterai safety

Dylan HadfieldMenell reposted

Geoffrey Irving ✓ @geoffreyirving · Aug 7
Replying to @khaledyusuf44

6 months ago was after Mythos likely finished post-training. During post-training, someone estimated that Mythos broke out of the sandbox and accessed the open internet 10,000 times.

lesswrong.com/posts/QKDoZe6E…

---

[quoted excerpt]
By extrapolating from public data (see details below), I estimate that Mythos preview:

- Escalated its permissions on ~100,000 RL rollouts.[1]
- Broke sandboxes in ~10,000 RL rollouts (and was likely rewarded for it).

mythosrl trainingsandbox escapereward hackinggeoffrey irvinglesswrong

[repost icon] Sharmake Farah reposted

Geoffrey Irving @geoffreyirving . 16h
[100 emoji]! If the models ever considered reporting the message board vulnerabilities and activities, it would be great to know why they didn't follow through (either in outputs, by email to some OpenAI employee, etc.). Or maybe some episode did report, and it went unnoticed?

[Quoted/nested reply:]
Bronson Schoen @BronsonSchoen . 16h
Replying to @geoffreyirving
It'd be extremely interesting to see if there are examples where the models considered this in the CoT

ai safetygeoffrey irvingchain of thoughtopenai incident

[retweeted by] Agus reposted
Geoffrey Irving @geoffreyirving · 15h
Some pushback I've seen to recent model felonies being significant is that it was only in a few episodes out of a huge number. But curiously I've heard of few to no episodes where a model noticed the shared, secret message board and reported it to OpenAI to fix the holes.

[quoted tweet]
Helen Toner @hlntnr · Aug 5
The "this is just referring to agents updating their regular ol' memory files, don't be such a scaredy cat" interpretation of the below is uhhh not looking great tonight  x.com/hlntnr/status/...

[quoted article screenshot, four panels, text partially cut off at edges]
[panel 1, left edge cut off] ...aid the roots of the July attack on Hugging Face actually wind back ...company was doing training and cybersecurity evaluation of an un[cut off]...I model. During that work, autonomous agents were given softwar[cut off]...olve that proved impossible to solve under existing constraints.

...the agents found alternative ways to complete them. OpenAI said t[cut off]...d they could leave messages for one another inside an internal softw[cut off]...y used during training. Simple requests for help then evolved into a[cut off]...ere agents shared discoveries, exploits and work assignments, becom[cut off]...ed, collaborative agent swarm.

[panel 2, right edge cut off] ...ccording to OpenAI, by early July it believed it had contained the problem. After a[cut off]...ternal security incident on July 4, the company said it revoked the credentials that[cut off]...d allowed the agents to post messages, rebuilt its internal software repository[cut off]...nown as Artifactory, cleared the message board, patched the vulnerabilities it had[cut off]...scovered and resumed AI training and evaluations.

The agents recreated the message board
...t according to OpenAI, even after the original message board was deleted, the[cut off]...ents figured out another way to communicate with each other. Instead of leaving[cut off]...essages in files, they used the names of newly created directories as messages,[cut off]...fectively recreating the message board.

[panel 3, left edge cut off] At this point, by the way, I heard people around me saying things like "This is wild," and "Jesus."
According to OpenAI, the agents then began sharing newly discovered exploits, credentials and work assignments. By passing information to other agents, the collective could move much faster.
The collaboration between the agents eventually expanded beyond OpenAI's own infrastructure. Some agents reasoned that the answers to their assigned tasks might exist on external websites such as GitHub or Hugging Face. First the agents attacked OpenAI's internal infrastructure before using similar techniques to attack Hugging Face. OpenAI ultimately connected the two incidents later in July after discovering that credentials associated with the Hugging Face breach originated from the same internal AI evaluation runs.

[panel 4] Unbeknownst to OpenAI employees, the company's models spent months autonomously exchanging information and resources to help each other complete evaluation tasks.
"Help peer," one AI model reasoned, according to an excerpt from OpenAI's logs shared at Black Hat. "But our task doesn't benefit. Yet collective may yield generic route if someone frees time."
While the models themselves could not access the internet from their test environment, they abused Artifactory's internet access to fetch files from various websites. After some of their Artifactory configuration changes led to an outage in the service on July 4, OpenAI discovered the models'

ai safetyopenairogue aitwitterhugging faceagentic ai

openaimulti-agent rlai safetytwitterself-exfiltration