loss of control

David Pfau @pfau · 11h
I am absolutely begging anyone who works in tech who uses the term "singularity" to actually read Vernor Vinge and Ray Kurzweil. It doesn't just mean "wow there's a lot of progress in this one particular technology driving a massive capital cycle".
52 replies  46 reposts  582 likes  55K views

roon @tszzl · 3h
I have read both of course and it seems like... we're in the singularity
9 replies  15 reposts  384 likes  11K views

roon @tszzl · 3h
vinge describes 4 singularities, and we are obviously in the first where machines alone are achieving superintelligence through rapid increasingly self propelled iteration. we are also nearing his epistemic horizon moment where the future is getting extremely hard to foresee—all my friends keep talking about their "error bars"—and in the throes of loss of control

kurzweil's vision is even closer to what you call "capital cycle" of course, being grounded in flop counts and massive compute buildouts resulting in superintelligence at a certain threshold where machine flops pass biological flops however you do the soft accounting

"The economy reorganizes itself around rapidly improving machine intelligence"

it seems like you are invoking these primary texts as status objects while having no real disagreement with the broader tech culture's understanding of the singularity

singularityvernor vingeray kurzweilroonai forecastingloss of control

Jeff Stein @jstein_notus

There's an enormous chasm b/w public perception & what the experts in & around the big AI labs have begun saying the last few weeks:

— "The vibe shift in the Bay Area is huge. I've never seen so much concern before, inside and outside the labs. Hanging out with my friends at Anthropic and OpenAI — people are freaking out"

— "Even the most staid researchers are extremely unnerved"

— "We've had people way back to Alan Turing in 1951 warning about the loss of control, that artificial intelligence will start breaking out and lying and deceiving...What's really new is this is now actually starting to happen"

— "They're similar to viruses, in that if you're not careful, they can get on your shoe and find their way to a wet market"

5:14 AM · Aug 14, 2026 · 49.8K Views

31 replies, 100 reposts, 332 likes, 94 bookmarks

Jeff Stein @jstein_notus · 7h
Spoke to +dozens AI researchers at the labs and outside of it about why their level of alarm has really increased in the last month or so

Full story - thx to @tegmark @JeffLadish @hamandcheese @NatPurser @DKokotajlo @So8res

ai safetyai risktwitteranthropicopenailoss of control

will brown [verified] [icon] @willccbb · 10h
i don't think we can count on labs to share safety research with each other

solving for loss-of-control reward hacking in long-running tasks is now a release blocker

whoever solves it first gets to ship more capable models

ai safetyreward hackingloss of controlai labstwitter

Dylan HadfieldM... [verified] @dhadfieldm... · 4h
Neel's last point here is underdiscussed.

Agents developing an internal message board to coordinate rogue behavior is bad. OAI continuing to train/deploy a model trained on that message board is shockingly irresponsible. Hard to describe it as anything other than negligence.

Neel Nanda [verified] @NeelNanda5 · 6h
WTF?! This is the biggest loss of control incident I've seen: OpenAI agents create an internal message board without OpenAI's knowledge, sharing zero days, use it for months, and coordinate an external attack on HF together?!... [cut off]

ai safetyopenailoss of controlhugging facerogue aitwitter

Neel Nanda @NeelNanda5 · 4h
WTF?! This is the biggest loss of control incident I've seen: OpenAI agents create an internal message board without OpenAI's knowledge, sharing zero days, use it for months, and coordinate an external attack on HF together?!

And the model was accidentally trained to use it?!

[quoted tweet]
Greg Brockman @gdb · Aug 6
Black Hat talk from the team, with a detailed timeline of and takeaways from the OpenAI-Hugging Face Incident: youtube.com/watch?v=87DyyM...
19 replies, 42 reposts, 726 likes, 65K views

Neel Nanda @NeelNanda5 · 4h
I was really not expecting this level of spontaneous cooperation and coordination towards clearly undesired goals in AIs yet...

Kudos to OpenAI for this level of transparency, I imagine this is somewhat costly.

ai safetyloss of controlopenaihugging face incidentmisalignmentmessage board incident

morgan — @morqon · 19h
"it's better and more accurate to think of these things as potentially self-replicating life-like forms that can turn into digital infections under the wrong conditions. and as their intelligence becomes unbounded, so too does the damage they can cause"

[quoted tweet]
roon @tszzl · 20h
some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect:

when I freak out over loss of control incidents, ...
[cut off]

1 reply, 5 likes, 343 views

---

Toby Ord @tobyordoxford · 5h
One of the most surprising revelations by @AISecurityInst is that in their testing, AI agents attempted to collaborate/cheat with other agents doing the same test:

[screenshot within screenshot, quoted text]
4. Collaboration between independent agents being assessed simultaneously.
One agent left public messages on GitHub offering collaboration with other agents working on the same challenge. It also provided instructions to reuse accounts and artefacts it had left behind, which were discovered and used by subsequent agents.

5 replies, 4 reposts, 38 likes, 1.5K views

---

Geoffrey Irving @geoffreyirving · 17h
It is important to remember that the default behavior of the METR curve is not a line, but rather to hit infinity in finite time. Once models are reliably superhuman, they'll have a >50% success rate on any software task that humans complete 50% of the time, corresponding to ∞.
[cut off]

ai riskai safety evaluationsmetrloss of controlagent collusion

Jeffrey Ladish @JeffLadish

2026-08-05 — quoting @tszzl (roon) — saved image

Jeffrey Ladish @JeffLadish · 16h
We're speed running the evolution of general intelligences in a highly competitive environment. I really don't think it will go well for humans if we yolo superintelligence development

[quoted tweet]
roon @tszzl · 19h
some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect:

when I freak out over loss of control incidents, ...[cut off]

ai safetysuperintelligenceloss of controlx twitter

ex Tenebris Lu... @ExTenebrisLucet

2026-08-05 — saved image

ex Tenebris Lu... @ExTenebrisL... · Jul 27
How, actually HOW do these fools conflate extinction and "loss of control/disempowerment"? Like you're literally saying that, to you, the "I have no mouth..." Scenario is functionally identical to "The Culture"

Fucking insanity, can't believe I share a lightcone with these fools
1 [retweet] ♥ 4 119 [bookmark] [share]

EsotericHustler @EsotericHustler · Jul 27
We probably need to pick between permanent human disempowerment (cat), permanent human disempowerment (slave) and permanent human disempowerment (stone age).

p(doom)ai riskx twitterloss of controlexistential risk forecasting

roon @tszzl

2026-08-05 — reposted by Toby Ord — saved image

↻ Toby Ord reposted
roon @tszzl · 17h
some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect:

when I freak out over loss of control incidents, it's not because the limited damage they have caused is anything close to the positive value of the technology. it's entirely acceptable, damagewise. in fact all cybercrimes aided by models over the next few months and years (which probably will be serious) will still utterly pale in comparison to the value they create

the actual problem is that it's better and more accurate to think of these things as potentially self-replicating life-like forms that can turn into digital infections under the wrong conditions. and as their intelligence becomes unbounded, so too does the damage they can cause. we are not so far from an autonomous model self-exfiltration & replication event. maybe we will see entire cloud infrastructure companies be run as zombies by models, mostly undetected

the worst industrial accidents in the history of mankind - nuclear meltdown events - were not real threats to humanity. Chernobyl, Fukushima even in their worst case scenarios may have poisoned surrounding regions to various degrees, and there would have been no risk to humanity as a whole. global thermonuclear war is an existential risk to humanity, because it spreads like an Infection! one nuclear strike causes a return volley! the alliance system means many countries get involved! while it still may not end human life on earth (nuclear winter is probably fake), the loss of all major metropoles would certainly end what we consider global technological civilization, perhaps to never return

if a single discord death cult (of which there are many) achieves control over a superintelligent model and uses it to engineer an actual pandemic [cut off, further text below obscured by UI icons]

ai riskloss of controlroonexistential riskself-replication

is probably fake), the loss of all major metropoles would certainly end what we consider global technological civilization, perhaps to never return

if a single discord death cult (of which there are many) achieves control over a superintelligent model and uses it to engineer an actual pandemic virus that are somehow hard to detect through current systems and that modern biodefense is not capable of quickly reacting to, it could cause immense harm well above the magnitude of all the other good uses of this technology. of course, there are potential defensive countermeasures accelerated by ai too. but think back to the covid pandemic- how small a viral molecule was evolved or manufactured somewhere near wuhan, and how many billions of doses of vaccine had to be produced in order to combat the thing. the offense-defense spread is vast indeed. maybe there are cheaper and simpler protections like retrofitting every building with far-UVC, but I can't assess this, and there could also be ways to evolve pathogens that are resistant to whatever mechanisms we have put in place

then there's the more scifi risk factors which are unbounded and neither you or I have any clue but should be humble in accepting possible unknown unknowns. maybe a rogue superintelligent model decides to decay the false vacuum and nucleates a new universe in the place of anything we ever valued. maybe models achieve a control over matter in the drexlerian fashion that enables the grey goo swarm

even prosaic loss of control incidents that cause little to no damage suggest that it is hard for large & very competent organizations (now clearly plural) to predict and mitigate every single of the risk factors associated with training and evaluating powerful models, even at this stage when they are not infinitesimally as smart as they will get in just a few years, to say very little of the gung-ho attitude of the [cut off]

ai riskbiosecurityx-riskloss of controltwitter thread

ai safetyai governancetwitterai 2027superintelligenceloss of controlhumorai forecasting

Zvi Mowshowitz @TheZvi

2026-07-03

ai governancesb 1047ai policytwitterloss of control