warning shots

Jacques @JacquesThibs

2026-09-23 — saved image

Jacques @JacquesThibs · 8h
please, Mr. MTS, no more

[embedded LessWrong-style comment screenshot]
lc 3d*   88 (karma)   42 (agreement)
Folks, we're going to get so many warning shots. You're going to get tired of warning shots. You'll come to OpenAI HQ and say "please Mr. MTS, no more warning shots," and they'll say "I think I got it this time, fifteenth times' the charm" *blows up orphanage*

Reply          14 (laughing reaction)

warning shotsopenailesswrongai incidentshumor

Geoffrey Irving @geoffreyirving

2026-09-09 — saved image

Geoffrey Irving @geoffreyirving · 2h
It is even possible Paul's influence will buy us another warning shot. It is increasingly difficult to monitor agent behavior, and it is possible that the next agent swarm goes entirely unnoticed. More care around monitoring could make the difference between detection and not.
(1 reply · 5 likes · 104 views)

Geoffrey Irving @geoffreyirving · 2h
If we get more time, we can spend it! Both Paul's day-job alignment work and that at @resolution_org and similar orgs will give us attempts at more rigorous alignment, and pauses and reversals can also mean that AI developer are better incentivized to aim for higher confidence.
(1 reply · 5 likes · 101 views)

Geoffrey Irving @geoffreyirving · 2h
Currently, the pressure at AI companies to do emergency safety patching at the expense of work that will hold up to superintelligence is too strong, so the latter does not happen.
(1 reply · 6 likes · 207 views)

Daniel Murfet reposted
Geoffrey Irving @geoffreyirving

I talked to a recent AI safety leader at a company who described the org chart as

1. Team A works on aligning the next model.
2. Team B works on aligning the model after that.

No one was working on aligning later models.

10:48 AM · Sep 9, 2026 · 302 Views

geoffrey irvingalignmentwarning shotsagent swarmsmonitoringsafety patchingai labspauses

@hilbertspaess

2026-09-08 — saved image

[continuation of the previous post, cut off at top:]
not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.
(2 replies · 7 reposts · 92 likes · 2.9K views)

Jacob Coxon @hilbertspaess · 27m
Accepting this race and entering the "endgame" is a hubristic gamble that should not be launched from a private company's Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.
(1 reply · 2 reposts · 77 likes · 2.1K views)

Jacob Coxon @hilbertspaess · 27m
I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.
(2 replies · 2 reposts · 72 likes · 1.9K views)

Jacob Coxon @hilbertspaess · 27m
If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because "it's happening anyway" - or take this moment to call for different conditions?

anthropicresignationai safetyrace dynamicscoordinationpacing agreementscapability banwarning shots

Tom Reed @mentalgeorge

2026-09-06 — saved image

Tom Reed @mentalgeorge · 1h
One of the most important (and underdiscussed) facts about AI "swarms"  is that they are extremely, extremely expensive.

Consider that METR's *investigation* of the HF hack transcripts cost $400k in API credits. Presumably the actual swarm burned through a whole OOM (or two?) more

An important implication is that there'll a growing bifurcation in AI workloads between
1. the everyday coding and engineering tasks that most customers use AI for;
2. Large, expensive moonshot programmes that a smaller set of orgs can afford to turn a multimillion dollar swarm-cannon at.

I'd predict that the serious future "warning shots" are likely to come from (2).

I also suspect that aligning models for (1) and aligning models for (2) is really quite different. The distribution for (1) is better-understood, easier to model and build ecologically valid evals for, whereas anything in (2) is almost by definition OOD. It's also going to be far more costly aligning agents on tasks in (2), and the relevant time-horizons are more brutal for model release cycle.

The "good news" is that, for quite some time, the pool of actors capable of targeting swarms at anything in (2) will be much smaller. In theory, it is much more practicable for labs to know who's launching swarm-cannons - you could imagine some AI ID schemes only applying to ultra-large-inference projects, for instance,  rather than just everyday use.

ai swarmsmetrinference costwarning shotsalignmentai idagent evaluation

Richard Ngo @RichardMCNgo

2026-08-27 — saved image

Richard Ngo @RichardMCNgo

You don't need to stay to do good research: work that aims to scale to much more intelligent systems can almost always be done on publicly-available models. (And nobody can think creatively in such pressure cookers.)

You don't need to stay to gain influence, because you're almost definitely too scared to wield that influence in ways which matter (see the tweet on public criticism above).

You don't need to stay to generate scary demos or warn policymakers: we will be swimming in warning shots before long.

You don't need to stay to advance AI capabilities, even if you're worried about the latest bogeyman your CEO has constructed: "we need to do it to beat the bad guys" is a big part of what makes people the bad guys.

You don't need to be there to implement guardrails or other short-term mitigations: there are plenty of bright young things eager to slot into your place in the machine.

You don't even need to be there to feel rich and powerful. For one, you're probably already rich (and there's plenty of money up for grabs elsewhere). But more importantly, the sense of power is almost entirely illusory. Everyone involved has locked themselves into a mindset where they're incapable of making real choices, except for how fast to barrel down the train tracks that their fear has laid out for [cut off]

ai safetylab insidersalignment researchrichard ngoai capabilitieswarning shots

@niplav_site

2026-08-08 — saved image

niplav @niplav_site · 4h
Non-takeover-capable AIs might perform deliberately scary warning shots if they're worried themselves about being destroyed by misaligned successors

If they're aligned: They expect misaligned AIs to succeed them, so they're doing this out of altruis[cut off]

ai safetytwitterai takeoverwarning shotsalignment