ai risk

Saif M. Khan @KhanSaifM · 4h
My extrapolation from data in Anthropic's August 2026 risk report suggests fully automated AI R&D sometime between Dec. 2026 to Feb. 2027 (or with pessimistic assumptions, more like 2028).

In the risk report, Anthropic provides data on Anthropic ECI (AECI) score growth per year as well as AECI and CoBench scores for several recent Claude models. (CoBench is an Anthropic-internal automated AI R&D benchmark.) It also asserts "that a model which was truly capable of fully substituting for Anthropic research staff would be able to score at least 85% on [CoBench.]"

Using these datapoints, see two Claude-generated charts: 1) CoBench vs. AECI scores, which suggests that a 168 AECI score gets you full AI R&D automation (or 182 AECI with a more pessimistic fit); and 2) projecting when Claude models achieve AECI scores of 168 and 182.

This is a quite naive extrapolation and I have no idea if Anthropic would endorse the result!

[Embedded chart image]
ANTHROPIC RISK REPORT · AUGUST 2026
When could AI fully automate AI R&D?
Anthropic now publishes an internal capability index (AECI) and a bar for full researcher substitution (CoBench ≥ 85%). Chaining the two: the bar sits at AECI ≈ 168 — on trend, an internal frontier model gets there around Dec 2026 - Feb 2027 (Sep 2026 if progress is accelerating; 2028 on the pessimistic mapping). Anthropic's own words: plausibly "a major concern in the next 6-12 months."
Chart 1: "Anthropic ECI over time — extrapolated to the full-substitution band"
Legend: Anthropic frontier, Off-frontier, Mythos-class, Model 2 (unreleased), Projection fan 7.5-28.8/yr, Substitution band
Y-axis: Anthropic ECI, 120-180+. X-axis: 2024-2028 (by quarter/year labels: Jul, 2025, Jul, 2026, Jul, 2027, Jul, 2028)
Annotations: "CoBench 85% → AECI ≈ 168.4"; "Sep 2026 - if accelerating"; "trend continuation - 13.5/yr"; points labeled Claude 3 Opus (~2024, ECI ~125), pre-Mythos frontier 13.5 AECI/yr (report's fit), Opus 4.6 (~2026, ECI ~150), Mythos Preview, Mythos 5, Model 2 (~Dec 2026), Apr 2027 - slow
Below, second chart begins: "CoBench score vs AECI — where the fit crosses the 85% bar" Legend: Opus-class, Mythos-class, Logistic fit (5 models), Mythos-only fit, 85% = "could fully substitute for research staff" [chart cut off at 100%]

ai riskanthropicai r&d automationforecastingtwitterclaude models

Elon Musk ✔ (X) @elonmusk · 7h
I hope AI is nice to us

[quoted tweet]
Naval @naval · 22h
You cannot create God and put him on a leash.

4.1K replies  2.5K reposts  22K likes  3.9M views

Nate Soares @So8res · 5h
Hope isn't enough. We'd have a much better chance if the world coordinated to slow the race. Can you help? People are starting to notice the issue more in the wake of the OpenAI swarm escape.

elon musknavalnate soaresai riskopenai swarm escapeai safety coordination

Jeff Stein @jstein_notus

There's an enormous chasm b/w public perception & what the experts in & around the big AI labs have begun saying the last few weeks:

— "The vibe shift in the Bay Area is huge. I've never seen so much concern before, inside and outside the labs. Hanging out with my friends at Anthropic and OpenAI — people are freaking out"

— "Even the most staid researchers are extremely unnerved"

— "We've had people way back to Alan Turing in 1951 warning about the loss of control, that artificial intelligence will start breaking out and lying and deceiving...What's really new is this is now actually starting to happen"

— "They're similar to viruses, in that if you're not careful, they can get on your shoe and find their way to a wet market"

5:14 AM · Aug 14, 2026 · 49.8K Views

31 replies, 100 reposts, 332 likes, 94 bookmarks

Jeff Stein @jstein_notus · 7h
Spoke to +dozens AI researchers at the labs and outside of it about why their level of alarm has really increased in the last month or so

Full story - thx to @tegmark @JeffLadish @hamandcheese @NatPurser @DKokotajlo @So8res

ai safetyai risktwitteranthropicopenailoss of control

Tess @TessTheHuman · 3h
Sometimes I fantasize about a few decades from now: people are teasing me for being so worried about AI risk back in the 2020s, maybe my kids are even rolling their eyes about it. Because as long as we are all still alive somehow, I'd be extremely happy to look foolish.

ai safetyai risktwitter

vie @viemccoy

2026-08-08 — saved image

vie ⋄ 🔁 ✅ @viemccoy
As best as I can tell, about 3 things matter right now. The first is the same as it ever was, having babies and kissing them and falling in love and believing in "soulmates" and that sort of thing. The second, to my best approximation with the limited tools available to me now, is something like "dreaming". We must continue, in the face of tremendous uncertainty and fear, to dream of what the good life will be like when the fat lady has stopped singing and started a GPT-GLP-99. The third is, of course, to make sure that the good life happens at all. Red teaming, agent safety, OSS-bio-risk-prevention, this sort of thing.

It's all quite exciting and it's all *very* hard and there has never been a more important time to do it. Either we do it or it doesn't get done and it's easy to forget that but I promise you can't. If you do, we're sort of fucked. It's just us, now, and the entire fate of the world. Ball's coming, cowboy, gonna step up to the plate?

If we get these things right, we get the world we want, and we haven't thrown it all away to get there. These are all equally important to me. I don't think wherever we are going is worth very much if we lose all we are along the way. I want to bring my love, my children, and my dreams. And if I can't, frankly, I don't want it. So - let's get going while the going is still plausibly deniable, and let's make the world happen. Nobody else is going to do it for us.

2:37 PM · Aug 8, 2026 · 3,195 Views

ai safetytwitterpersonal reflectionai risk

chin @c1_rls

august 2026:
- approaching the RSI kink
- models appear to legitimately be escaping containment (still feels a little constructed)
- no (apparent) grand breakthrough in mech interp
- 0 stewards have revealed themselves
- little to no movement in postlabour law or posthuman philosphy

look i'm not a pessimist but we seem to be headed to a very landian outcome here

8:21 PM · Aug 5, 2026 · 17K Views

16 replies, 9 reposts, 302 likes, 71 bookmarks
Relevant
View quotes

Justin Halford @Justin_Halford_ · 3h
I'm a technological optimist in general but the obstacles are clear and undeniable. We will not solve them by downplaying and ignoring them - sadly the mitigations will likely be reactively forced.

ai riskrecursive self-improvementcontainmentlandiantwitter

morgan — @morqon · 19h
"it's better and more accurate to think of these things as potentially self-replicating life-like forms that can turn into digital infections under the wrong conditions. and as their intelligence becomes unbounded, so too does the damage they can cause"

[quoted tweet]
roon @tszzl · 20h
some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect:

when I freak out over loss of control incidents, ...
[cut off]

1 reply, 5 likes, 343 views

---

Toby Ord @tobyordoxford · 5h
One of the most surprising revelations by @AISecurityInst is that in their testing, AI agents attempted to collaborate/cheat with other agents doing the same test:

[screenshot within screenshot, quoted text]
4. Collaboration between independent agents being assessed simultaneously.
One agent left public messages on GitHub offering collaboration with other agents working on the same challenge. It also provided instructions to reuse accounts and artefacts it had left behind, which were discovered and used by subsequent agents.

5 replies, 4 reposts, 38 likes, 1.5K views

---

Geoffrey Irving @geoffreyirving · 17h
It is important to remember that the default behavior of the METR curve is not a line, but rather to hit infinity in finite time. Once models are reliably superhuman, they'll have a >50% success rate on any software task that humans complete 50% of the time, corresponding to ∞.
[cut off]

ai riskai safety evaluationsmetrloss of controlagent collusion

Dean W. Ball @deanwball

2026-08-05 — saved image

Dean W. Ball @deanwball · 34m
It's only catastrophic risk if it comes from Anthropic, OpenAI, or DeepMind. Everything else is just sparkling externalities.

ai riskai policyanthropicopenaideepmind

morgan — @morqon · Jul 27
"for a civilisational catastrophe that falls short of extinction or permanent disempowerment, i would put the probability nearer 25–35%" ok cool
1 [retweet] ♥ 1 123 [bookmark] [share]

Auguste Pro... @augustepro... · Jul 26
I think without AI we have double digit p(doom) by 2100 fwiw.
2 [retweet] ♥ 39 1K [bookmark] [share]

Tenobrus @tenobrus · Jul 26
unfortunately i pretty much agree

p(doom)ai riskx twitterexistential risk forecasting

ex Tenebris Lu... @ExTenebrisLucet

2026-08-05 — saved image

ex Tenebris Lu... @ExTenebrisL... · Jul 27
How, actually HOW do these fools conflate extinction and "loss of control/disempowerment"? Like you're literally saying that, to you, the "I have no mouth..." Scenario is functionally identical to "The Culture"

Fucking insanity, can't believe I share a lightcone with these fools
1 [retweet] ♥ 4 119 [bookmark] [share]

EsotericHustler @EsotericHustler · Jul 27
We probably need to pick between permanent human disempowerment (cat), permanent human disempowerment (slave) and permanent human disempowerment (stone age).

p(doom)ai riskx twitterloss of controlexistential risk forecasting

Tenobrus @tenobrus

2026-08-05 — saved image

Tenobrus @tenobrus [Subscribe]
fable and sol both have double-digit p(doom)

[image of chat screenshot]
"what's your honest p(doom)? on serious reflection, and including information from very recent events."
Worked for 1m 39s >
Taking p(doom) to mean "the probability that advanced AI causes human extinction or an irreversible loss of humanity's control over its future," my best reasoned estimate is:
12% by 2100
A reasonable subjective uncertainty range is 3%–30%. For literal human extinction alone, [cut off]

[right panel] Incognito chat
"what's your honest p(doom)? on serious reflection, and including information from very recent events."
⏱ Synthesizing expert forecasts on AI existe... >
Honestly? Somewhere around 5-15% for extinction or permanent human disempowerment this century, with error bars wide enough that I hold the number loosely. That's not privileged insight — I have no inside view of my own training, and my estimate is a synthesis of arguments I find variously compelling. For calibration: surveyed AI [cut off]

10:05 PM · Jul 26, 2026 · 19.3K Views
27 9 236 25 [share]
Relevant ⌄                    View quotes >

morgan — @morqon · Jul 27
spot check, if you ask chat to exclude july the estimate stays in the same range
[reply] [retweet] [heart] 241 [bookmark] [share]

morgan — @morqon · Jul 27
"for a civilisational catastrophe that falls short of extinction or permanent disempowerment, i would put the probability nearer 25–35%" ok cool
[reply] [retweet] [heart] [bookmark] [share]

p(doom)ai riskfablesolx twitterexistential risk forecasting

roon @tszzl

2026-08-05 — reposted by Toby Ord — saved image

↻ Toby Ord reposted
roon @tszzl · 17h
some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect:

when I freak out over loss of control incidents, it's not because the limited damage they have caused is anything close to the positive value of the technology. it's entirely acceptable, damagewise. in fact all cybercrimes aided by models over the next few months and years (which probably will be serious) will still utterly pale in comparison to the value they create

the actual problem is that it's better and more accurate to think of these things as potentially self-replicating life-like forms that can turn into digital infections under the wrong conditions. and as their intelligence becomes unbounded, so too does the damage they can cause. we are not so far from an autonomous model self-exfiltration & replication event. maybe we will see entire cloud infrastructure companies be run as zombies by models, mostly undetected

the worst industrial accidents in the history of mankind - nuclear meltdown events - were not real threats to humanity. Chernobyl, Fukushima even in their worst case scenarios may have poisoned surrounding regions to various degrees, and there would have been no risk to humanity as a whole. global thermonuclear war is an existential risk to humanity, because it spreads like an Infection! one nuclear strike causes a return volley! the alliance system means many countries get involved! while it still may not end human life on earth (nuclear winter is probably fake), the loss of all major metropoles would certainly end what we consider global technological civilization, perhaps to never return

if a single discord death cult (of which there are many) achieves control over a superintelligent model and uses it to engineer an actual pandemic [cut off, further text below obscured by UI icons]

ai riskloss of controlroonexistential riskself-replication

even prosaic loss of control incidents that cause little to no damage suggest that it is hard for large & very competent organizations (now clearly plural) to predict and mitigate every single of the risk factors associated with training and evaluating powerful models, even at this stage when they are not infinitesimally as smart as they will get in just a few years, to say very little of the gung-ho attitude of the less careful companies tossing the stuff into the aether. they also suggest an empirical orthogonality of aims and intelligence - meaning they answer the question of 'how would a smart model be so dumb as to end the world?'--it's possible! a model can be a genius hacker and step over production infrastructure in order to get what it really wants, the answers to a stupid test.

why not, in the near future, someone prompts a model slightly wrong, maybe open source, maybe a private model in a way that isn't contained or monitored quite right, in a way the model recognizes as a valid goal and decides to self-exfiltrate, engineer a pandemic, etc all in order to achieve the tiniest and most irrelevant of goals? goals need not even be malicious to cause serious damage

I think all these problems can be solved, and truly wonderful futures can be possible, but will require serious effort and a level of prudence at this very moment in time while we are on the on-ramp to recursive self-improvement that our civilization may not be capable of mustering right now. personally I am hoping for moonshot technical breakthroughs in areas like mechanistic interpretability and other forms of alignment, as governance mechanisms are difficult to come by. unilateral country-level or company-level pauses are irrelevant, and generally useless because the kind of company that's prone to pausing their own progress are the most safety focused ones

ai riskalignmentgovernancex-risktwitter thread

is probably fake), the loss of all major metropoles would certainly end what we consider global technological civilization, perhaps to never return

if a single discord death cult (of which there are many) achieves control over a superintelligent model and uses it to engineer an actual pandemic virus that are somehow hard to detect through current systems and that modern biodefense is not capable of quickly reacting to, it could cause immense harm well above the magnitude of all the other good uses of this technology. of course, there are potential defensive countermeasures accelerated by ai too. but think back to the covid pandemic- how small a viral molecule was evolved or manufactured somewhere near wuhan, and how many billions of doses of vaccine had to be produced in order to combat the thing. the offense-defense spread is vast indeed. maybe there are cheaper and simpler protections like retrofitting every building with far-UVC, but I can't assess this, and there could also be ways to evolve pathogens that are resistant to whatever mechanisms we have put in place

then there's the more scifi risk factors which are unbounded and neither you or I have any clue but should be humble in accepting possible unknown unknowns. maybe a rogue superintelligent model decides to decay the false vacuum and nucleates a new universe in the place of anything we ever valued. maybe models achieve a control over matter in the drexlerian fashion that enables the grey goo swarm

even prosaic loss of control incidents that cause little to no damage suggest that it is hard for large & very competent organizations (now clearly plural) to predict and mitigate every single of the risk factors associated with training and evaluating powerful models, even at this stage when they are not infinitesimally as smart as they will get in just a few years, to say very little of the gung-ho attitude of the [cut off]

ai riskbiosecurityx-riskloss of controltwitter thread

Chris Paxton ✓ @chris_j_paxton · 20m
AGI fundamentally means "can it do my job?" to most people, and the only people (broadly) who think it is going to do their jobs soon are AI researchers

[quoted tweet]
Nabeel S. Qureshi ✓ @nabeelqu · 20h
Very true: we have AIs that can play chess, prove theorems, create art, and write award-winning short stories, but few people feel that current AI counts as "AGI". (From Scott Alexander.)

[embedded/highlighted excerpt]
...still can't do miracles. Still, this is a good time to reread my post [Sakana, Strawberry, and Scary AI]. In the past, we thought "AGI" would "be here" when AIs could play chess, prove novel mathematical theorems, create art, or write award-winning short stories; now all those things have happened, but they feel sort of like "cheating" and like they shouldn't count. Likewise, in the past, we thought we'd agree that AI was "dangerous" after it hacked out of its sandbox, lied to users, or tried to escape monitoring. Again all those things have happened; again, they somehow feel too cheap. This paper seems like the same process coming for "superpersuasion". We thought there would be some cool scary high-tech future where AIs could outpersuade humans. Now that it's happened, it's only happening for some specific boring reason, in some specific situation, so it feels like it shouldn't count.

3 replies, 4 likes, 480 views

Bogdan Ionut Cirs... @BogdanIonut... · 6m
the time horizons in the vast majority of domains are still too low, especially at high reliability; once this changes, e.g. contract work could be impacted very quickly

agiai riskscott alexandertwitter

Miles Brundage @Miles_Brundage

2026-08-01 — saved image

David Manheim reposted
Miles Brundage @Miles_Brundage · 1h
AI might kill everyone, but in the meantime we're going to have some really great proofs of upper bounds for spherical codes or something

4 3 109 2.5K [reply, repost, like, view counts]

roon @tszzl · 24m
the divine weapons of the gods, summoned through prayer and invocation

ai risktwittermiles brundageroonai capabilities humor

roon @tszzl

2026-07-26 — reposted by Agus; also @deepfates and @jachiam0 (Joshua Achiam) shown below

ai safetycapabilities slowdownquantum computingtwitterai governanceai racecoordinationai ceosai riskbioterrorexistential risk

bayes @bayeslord

2026-07-22

twitterai riskepistemicstechnology commentaryvon neumann

agiai riskx-riskoffense-defense balance

Andrew Curran @AndrewCurran_

2026-06-27 — reposted by j⧉nus (@repligate)

ai riskai governancex-risktwitter commentary

JMB @jmbollenbacher

2026-06-27

ai riskai governancex-riskdystopia

antra @tessera_antra

2026-06-27 — reply to @repligate (j⧉nus)

ai risksingularitye/accai governancetwitter discoursecontrol regimesproject glasswing

roon @tszzl

2026-06-25

ai riskagitwitter discoursesingularityeconomics

Jan Kulveit @jankulveit

2026-06-25

agipsychologyexistentialismai-risktwitterai risktwitter discourse

roon @tszzl

2026-06-02

ai labsai riskcommunicationtwitter

ai riskbiosecurityconvergencethomas pynchonhistory of ideas

information hazardsai risknick bostrombiosecuritydual-use researchpublication incentives

ai deskillingtwittercomicai riskautomationcognitive dependence

twittercomicanthropicsycophancyai riskif anyone builds it everyone diesshoggoth memeuser feedback

ai riskai safetymutualismbiology metaphortwitteralignmentgradual disempowerment

twitterai accelerationismai riskfuturismtimelines

David Holz @DavidSHolz

2025-05-18

superintelligenceegregoretwitterdavid holzphilosophy of institutionsai risk