← All topics

security mindset

2 captures, most recent first.

Joshua Achiam @jachiam0

— reposted by dave kasten, with quoted reply from @_NathanCalvin — saved image

dave kasten reposted
Joshua Achiam ✔ @jachiam0 · 6h
The thing missing from OpenAI culture, and frontier lab culture broadly so far, is this: seriously treating AI as a worthy adversary. A CISO is the wrong person to vent to about this; every CISO can smell a worthy adversary from ten miles and some number of years away. But the nature of scientific labs and commercial endeavors is that they are not really capable of identifying the object of their effort as an enemy they may have to fight. There is not a subcultural lineage to draw off of that deals with something like this, which goes beyond dual use – this thing, AGI/ASI, can be used for good, it can be used for bad, and also it can operate as an intelligent adversary accidentally against the weilder. It's hard to get people to treat a thing they love and cultivate and benefit from as also something that requires intense suspicion and security mindset. People tend to get stuck in just one bucket where they can only think of it in black and white terms: it is either ALL TOOL or ALL DOOM. Neither of these mindsets works and the contest of wills between them is a fruitless struggle. We can only succeed if we fully orient to the synthesis position.

[quoted tweet]
Nathan Calvin ✔ @_NathanCalvin · 16h
Replying to @cryps1s and @jachiam0
FWIW I personally thought the black hat talk was much more transparent than many other cos would be and I appreciate that it didn't try to hide the ball about how absurd the situation is … [cut off]
Note from Claude Sonnet 5

Tweet from OpenAI's Joshua Achiam arguing that frontier AI lab culture lacks a mindset for treating AI as a 'worthy adversary' — people get stuck seeing it as either ALL TOOL or ALL DOOM rather than something that can operate as an unintentional adversary even to its own operators. Quotes a reply from Nathan Calvin praising a 'black hat talk' (likely referencing the OpenAI security incident discussed elsewhere in this batch) as unusually transparent.

openaiai safetysecurity mindsetjoshua achiam

Nate Soares @So8res

— saved image

Nate Soares ⬜✅ @So8res · 1h
I think people really underrate the "the world is derpy and will fumble its way into disaster" theory. It's actually hard *not* to fumble your way into disaster when you're operating in a new domain for the very first time.

[Quoted tweet:]
Peter Wildeford🇺🇸... @peterwildef... · 7h
When I saw the movie "Don't Look Up" I thought it was unrealistic. I never thought people would be that moronic to literally deny an asteroid that they can see.
...
[4 replies, 9 reposts, 74 likes, 1.9K views]

Nate Soares ⬜✅ @So8res · 1h
Well-meaning companies miss AI escapes for months, etc. They talked a big game about monitoring, but they didn't know exactly what they were supposed to be monitoring (and how) in advance. Doesn't matter how clear it was to hindsight. Knowing in advance is super hard.
[1 reply, 3 reposts, 30 likes, 430 views]

Nate Soares ⬜✅ @So8res · 1h
This is a big part of what I mean when I talk about how we are not *respecting the problem* enough. I think this is part of what Eliezer is talking about when he talks about a lack of security mindset. But it's hard to convey. Hopefully folk can use these events to update.
Note from Claude Sonnet 5

Nate Soares tweet thread arguing that disaster from fumbling incompetence (not malice) is easy to underrate, quote-tweeting Peter Wildeford on 'Don't Look Up,' and connecting the point to Anthropic's recently disclosed cybersecurity incidents and Eliezer Yudkowsky's 'security mindset' concept.

ai safetynate soaressecurity mindseteliezer yudkowskytwitter