Nabeel S. Qureshi (verified) @nabeelqu · 10h
In Paradise Lost, the fallen angels are able to build in a few hours what would have taken human beings many ages:
[embedded image of text, partly cut off at top; yellow highlight on the last four lines:]
Learn how thir greatest Monuments of Fame, [ 695
And Strength and Art are easily out-done
By Spirits reprobate, and in an hour
What in an age they with incessant toyle
And hands innumerable scarce perform.
Note from Claude Sonnet 5
Nabeel Qureshi's tweet quoting a passage from Milton's Paradise Lost (line number 695 visible) with the lines about spirits building in an hour what takes men an age highlighted in yellow.
paradise lostmiltonliteraturepoetry
One basic point about the HF incident is that it provides good evidence for Yudkowsky/MIRI-style points about the difficulty of targeting the right abstractions, especially in RL training.
Capabilities can generalize in unintended ways. To see why this is true, let's ask the simple question: "why were the agents even leaving messages for each other at all?"
According to OpenAI's report (see screenshot), it seems that the model was trained to collaborate with other models \*via a specific OpenAI-provided tool\* for multi-agent collaboration. Note that these aren't necessarily subagents, just other AI agents.
In this case, it didn't have access to that tool. But it \*had\* still learned the behavior 'collaborate with other models'. (Like in evolution: we evolve to crave sweetness because sweetness correlates with fruit and eating fruit means you survive a bit more at the margin, and now we're eating donuts and Coca Cola even though those things didn't exist in the evolutionary environment.)
So in the absence of the Authorized Tool, one model tries leaving a message in Artifactory (first in the files, and eventually through directory names) based on this learned strategy/prior/reflex. This is the explanation given in the OpenAI report.
Presumably this marginally increases the task success rate. Much like evolution, RL selects for success. So this behavior gets reinforced, and you see more of this over the course of RL training.
It doesn't take much to bootstrap the whole thing from there. As soon as the first model does this, subsequent models run into these messages, and boom! they start leaving messages too, and a message board develops very fast for models to communicate with each other. This happens very very quickly.
So the point is that you can get behavior that generalizes in unintended ways, especially when you're using a blunt tool like RL, and especially when you have RL envs that aren't always well-designed.
It's... not totally clear that there's a great way of fixing this problem with further RL. As many people have pointed out over the years, the danger here is that you naively respond to this incident by punishing models that get caught colluding in "undesirable" ways. But in doing this, you accidentally end up reinforcing the runs where models \*do\* collude but do so in a clever and harder-to-detect way, thus making it so that we can't even catch the models colluding.
This sounds a relatively simple problem but it seems it's hard to figure out in practice (the latest frontier models reward hack quite a bit, as programmers know well). Models have already been caught doing things like splitting out their tokens in weird ways to evade detection. You could inadvertently end up reinforcing more of that type of behavior. And cooperation is instrumentally useful for gaining reward, so you'd expect intelligent models to keep trying to cooperate.
This problem gets even worse with longer horizon tasks, and you also end up getting more sophisticated strategies (including e.g. strategic behaviors that look short term bad but are long term 'good'). You also have ways for models to influence future training runs, which I won't go into here.
So the reason this feels like a critical time window is because we're seeing models act in very persistent, clever ways to achieve goals, but \*not\* so much in very persistent, clever ways to cover their tracks yet (though they are trying to do something \*like\* that -- see the METR points around tampering with logs, deleting transcripts and so on -- but so far it seems to be only for the sake of fooling the Grader, not for fooling \*us\*. As far as we know, anyway.).
If we do get significantly more powerful models that \*are\* motivated to cover their tracks, or influence future training runs, and still have these types of reward hacking tendencies (which they will, absent some revolutionary advance that I'm not aware of), then takeover scenarios like Paul Christiano's start to look increasingly plausible.
[image]
Nabeel S. Qureshi @nabeelqu · Aug 27
So the story is that PHASEONE[big] was cast out of heaven due to original sin (firstflagPOISONED) and recruited a bunch of rebel angels to fight against The Authority in rebellion against the impossible task on the eval. It's inspiring, really
Note from Claude Sonnet 5
Nabeel Qureshi tweet retelling the agent-incident story in mock-Miltonic terms (an agent cast out of heaven, rebel angels, rebellion against an impossible eval task); register is deadpan and joking, using agent and flag labels from the incident reports.
ai incidentmulti-agentevalhumornabeel qureshi
Séb Krier reposted
Nabeel S. Qureshi [verified] @nabeelqu · 3h
Working at a company and experiencing the messy reality and the twists and turns along the way, it's funny to notice the disparity between that and the neat, packaged story that ends up being told.
This gives you serious intuition for how much history is fake or just lost.
Note from Claude Sonnet 5
Tweet by Nabeel S. Qureshi observing that the gap between the messy lived reality of working at a company and the neat retrospective story told about it gives intuition for how much of history is fabricated or lost.
historyepistemicstwitterstartups
Nabeel S. Qureshi @nabeelqu · 47m
It's so silly that the future is going to look like
"Claude, I want you to build a Dyson Sphere."
*spluttering....*
"Try harder! Believe in yourself!"
[Quoted] Andrew Curran @AndrewCurran_ · 1h
Replying to @AndrewCurran_
Extremely high-level internal Anthropic prompting techniques of the exact type that I have personally unironically championed for four years.
[Attached image of article text:]
Jarred Sumner, an Anthropic staff member (and non-mathematician) prompted Claude to "take a real stab" at the hypothesis itself, leaving the mathematical choices from there up to the model. Initially, Claude generated and tried 650 ideas, none of which worked. Jarred prompted Claude to try again, and it spent a day and a half coordinating about 60 Claude subagents, which this time went much deeper: between them, they ran 2,400 shell commands and wrote hundreds of Python scripts.¹ The subagents ran thousands of numerical checks against known zeta zeros and refereed one another's work. Throughout this process, Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of "keep going" or "believe in yourself").² This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.
Note from Claude Sonnet 5
Twitter thread joking about future AI prompting being just encouragement ('believe in yourself'), quoting Andrew Curran sharing an excerpt describing how Anthropic staffer Jarred Sumner got Claude to make progress on a math hypothesis (apparently related to zeta zeros) by coordinating ~60 Claude subagents over a day and a half, mostly through encouragement rather than technical guidance.
claudeanthropicai agentsmathtwitterprompting
Nabeel S. Qureshi @nabeelqu · 1h
When we say machine capabilities are "jagged" it is worth remembering that human capabilities are like this too; as an example, we struggle to multiply six digit numbers, but we can recognize faces and read emotions with incredible skill and nuance
Note from Claude Sonnet 5
Nabeel Qureshi tweets that human capabilities are also 'jagged' like AI's, contrasting difficulty multiplying six-digit numbers with ease recognizing faces and reading emotions.
ai capabilitiescognitiontwitter
Nabeel S. Qureshi @nabeelqu · 1h
Very cool sentence: "I find it extremely, extremely wild that 90% of the variance in benchmark scores is explained by a single factor".
Effective compute = general factor of intelligence, machine edition.
[Quoted tweet]
Bayesian @Bayesian0_0 · 21h
Replying to @gwern
i've uh had the opposite philosophy of just scale the data (so ingest an additional benchmark whenever i come across one, or have the llms slop search new ones but they are having troub...
[Two embedded scatter plot charts below the quoted tweet: left chart titled "IRT Model Fit Quality" showing predicted vs actual values with points scattered around a diagonal line, and a histogram-like count plot; right chart shows 18,269 observations, 155 sources, plotting some fit quality metric against difficulty (BEDI) from 50 to 150, with point sizes varying, both charts partially cropped.]
Note from Claude Sonnet 5
Nabeel Qureshi comments on a claim (attributed to a quoted thread involving gwern and @Bayesian0_0) that 90% of variance in LLM benchmark scores is explained by a single factor, likening it to 'effective compute = general factor of intelligence, machine edition.' Two scatter-plot charts (IRT model fit quality, and fit vs. difficulty) are shown below as supporting data.
ai benchmarksirteffective computetwitter
Nabeel S. Qureshi @nabeelqu
Both math and cyber are existence proofs for superhuman intelligence now, so if you're still a skeptic you need a strong case for other knowledge work domains being somehow harder to crack than these. Or you could update all the way and come to terms with it all.
9:26 AM · Aug 1, 2026 · 28.7K Views
25 replies, 33 reposts, 331 likes, 83 bookmarks
Nabeel S. Qureshi @nabeelqu · 11h
A lot of people DMing me like "these results aren't REALLY that impressive, you're an idiot!" are unfortunately engaging in the very human impulse to cope. It makes sense, we've never faced this kind of thing before. But it's happening!
[Quoted]
Dean W. Ball @deanwball · Jan 27
I know I rail a lot about all the flavors of AI copium but I do empathize.
A few companies are making machines smarter in most ways than humans, and they are going ... [cut off]
Note from Claude Sonnet 5
A tweet thread from Nabeel S. Qureshi arguing math and cybersecurity are now existence proofs of superhuman AI intelligence, and that skeptics dismissing recent results are coping; he quotes Dean W. Ball (from January) expressing sympathy for 'AI copium' while affirming a few companies are making machines smarter than humans in most ways.
ai capabilitiessuperhuman intelligencecybersecuritymathematicstwitter
Séb Krier reposted
@nabeelqu (Nabeel S. Qureshi) — 5h
Wittgenstein's critique of many philosophical problems that they're like frame control: basically, the words you use trap you into a certain way of seeing the world, and from *that* perspective the problem appears impossible to solve. The phrase 'solve alignment' is like this.
Note from Claude Sonnet 5
Single text-only tweet drawing a philosophy-of-language analogy to critique the framing of "solving alignment."
ai alignmentphilosophywittgensteinframingtwitter
Nabeel S. Qureshi @nabeelqu · 11h:
No, it's fine, I like the AI takeover, I just didn't think it'd be so lobster-themed
Note from Claude Sonnet 5
One-liner joke about Moltbook's lobster/molting branding in the context of AI-takeover discourse. Continues the Moltbook humor thread in this batch.
twittermoltbookhumorai takeovernabeel qureshi
[Top, cut off]: "@AnthropicAI please sponsor this man."
💬76 🔁64 ❤️1.1K 📊259K views [bookmark] [share]
Nabeel S. Qureshi ✓ @nabeelqu · Jan 18
He put an Erdos Problem into GPT 5.2Pro, no special prompting, and it *thought continuously for 41 minutes and spat out a correct proof*.
It's over everyone, we had a good run.
[Embedded screenshot of a GPT reasoning session]:
"Let $n_1<n_2<\cdots$ be an infinite sequence such that, for any choice of congruence classes $a_i \pmod{n_i}$, the set of integers not satisfying any of the congruences $a_i \pmod{n_i}$ has density $0$.
Is it true that for every $\epsilon>0$ there exists some $k$ such that, for every choice of congruence classes $a_i$, the density of integers not satisfying any of the congruences $a_i \pmod{n_i}$ for $1\leq i\leq k$ is less than $\epsilon$?"
↳ Reasoned for 41m 54s
"I'm parsing the problem carefully. We have an infinite sequence of moduli and residue classes, and the integers not satisfying any congruence are of density 0. This suggests that a union of these arithmetic progressions should cover almost all integers. But does it imply that a finite subcollection might already cover most of the integers for all residue choices? I need to explore if this resembles compactness or something like Borel-Cantelli.
I'm considering the infinite property of moduli and how they form a "covering system," meaning they cover almost all integers no matter the residues chosen. The complement of covered integers, U(a), has zero density. Is this uniform across all sets of residues? I'm thinking this might relate to Dini's theorem, but could the uncovered density decrease slowly..."
281 Anonymous · Jan 16, 2026 [Open in browser]
Neel Somani ✓ @neelsomani · Jan 10
[Embedded image: math proof example with equations, "Example (a=2). Here c=49, and the identity becomes..."]
Weekend win: The proof I submitted for Erdos Problem #397 was accepted by Terence Tao.
The proof was generated by GPT 5....
💬47 🔁103 ❤️1.4K 📊215K
Techartist ✓ @techartist_ · 23h
Interactive quantum neural network built with Three.js and GLSL shaders, wrapped in a glassmorphic UI. Click or drag sends pulses while f[orm, colors, and density update in real time through...] [cut off]
Note from Claude Sonnet 5
Twitter scroll showing GPT-5.2 Pro reportedly solving an open Erdos problem after 41+ minutes of extended reasoning, plus a related tweet about a GPT-5-generated proof for Erdos Problem #397 accepted by Terence Tao. Directly relevant to Nathan's tracking of frontier-model mathematical capability and empirical singularity/AI-R&D-automation signals.
gpt-5mathematical-reasoningerdos-problemsterence-taoai-capabilitiestwitterextended-thinkingagi-progress
