Joshua Achiam [verified] @jachiam0 · 2h
Contrarian take: people are fixated on the "model used a message board to coordinate across instances" point and I think this is the wrong thing. Models externalizing memory, skills, context, etc is a useful design principle and we should assume they are doing this in the future. There is no version of the AGI/ASI future where models coordinating across instances via message boards or coded messages doesn't happen. The question is really whether the models are aligned, monitorable, and monitored. Also, whether there is sufficient test time compute allocated to discovering whether the models coordinating in the wild are remaining faithfully aligned to human interests and directives - which will (and here is where I will understate the key strategic insight, but please understand that this is the most important thing I am writing here) mean allocating more compute to monitoring than is allocated for practical usage.
Note from Claude Sonnet 5
A tweet by Joshua Achiam (OpenAI) responding to the same 'model used a message board to coordinate across instances' incident referenced elsewhere in this batch, arguing the real issue is alignment/monitorability rather than the coordination method itself, and that monitoring compute may need to exceed usage compute.
Joshua Achiam [verified] @jachiam0 · 22h
Something I really like about Three Body Problem is that it tries to extrapolate out the asymptotic dynamics of the alignment between civilizations over all of time and space in the universe. I don't feel it gives the definitive answer to the question of how alien species would interact because it remains unclear what types of interests alien species might realistically have. Are there many species-level reward functions compatible with the development of high technology or only a few? That is to say - are humans so far actually typical of species that exist in the universe and that can attain spacefaeing status? Or are multiple civilization/species types normal? This question feels especially important when trying to understand the long-term alignment dynamics of ASI and the types of civilizations that may exist in the far future after one or more ASIs enter a seed-spreading phase.
Note from Claude Sonnet 5
A tweet by Joshua Achiam (OpenAI) reflecting on the novel Three Body Problem as a lens for thinking about long-term alignment dynamics between civilizations/species and ASI seed-spreading scenarios.
Joshua Achiam @jachiam0 · 5h
Related to some of my earlier posts about RSI and threat models: I believe a huge strategic error is made when people model an ASI as an infinitely powerful and insurmountable threat. We should model, with more rigor, what types of adversarial AI we willl likely face, what the [Show more]
12 replies, 8 reposts, 90 likes, 4.5K views
Aaron Scher @aaronscher · 4h
Don't mistake a hole in your world map for a hole in everybody else's map. There exists some threat modeling like you're describing, albeit not a lot. E.g., alignmentforum.org/posts/LFNXiQuG..., lesswrong.com/posts/9YCJZBtq...
[Embedded link card: alignmentforum.org — "What does it take to defend the world against out-of-control AGIs..." with a preview image listing steps like "Tech company gives everyone API access to the AGI", "Tech company posts AGI.exe on its website as a free download", "Tech company publishes the recipe for rolling your own AGI.exe", "Too open! — some careless actor makes an out-of-control power-se..."]
1 reply, 16 likes, 234 views
Herbie Bradley @herbiebradley · 3h
this definitely is in the direction Joshua is proposing, but notable that in the list of 10 examples it contains many lines such as
> Before that process is finished, a different tech company accidentally makes an out-of-control AGI, which promptly exploits the not-yet-patched systems to trigger all-out nuclear war.
which would seem to be assuming the "insurmountable threat" ASI as a starting point
Note from Claude Sonnet 5
A three-way X thread on AI threat modeling: Joshua Achiam argues people wrongly model ASI as an infinitely powerful insurmountable threat and calls for more rigorous adversarial-AI threat modeling; Aaron Scher pushes back with links to existing alignment-forum/lesswrong threat-modeling posts; Herbie Bradley notes those examples still assume an 'insurmountable threat' ASI as a premise (quoting a scenario where a tech company's out-of-control AGI triggers nuclear war).
davidad @davidad · 2h
if your definition of "AGI" is "better at most tasks than per-task expert humans" (back in the day, we used to call this "ASI"), that is coming next quarter
[quoted tweet]
Bayesian @Bayesian0_0 · Aug 1
Fun fact: Across 44 benchmarks that have a "Human baseline", the human baseline BECI (a personal replication of the Epoch Capabilities Index) comes out at 166.7, which projections say will be beat by AI models around october 2026!
[embedded chart: 'Human baseline on the BECI scale (pooled human rows scored against frozen benchmark parameters; human data never enters the fit)'. Scatter plot, x-axis 'Release date' 2023-01 to 2026-07+, y-axis 'BECI' 60-160+. Legend: Models (grey dots), Model frontier (blue step line), Frontier trend (dotted line), Human baseline pooled (red horizontal band ~166.7). Annotation: 'trend crossing ~2026-10-16' where the frontier trend dotted line meets the red human baseline band.]
Note from Claude Sonnet 5
X thread: davidad comments on Bayesian's (@Bayesian0_0) chart showing AI model capability (a personal replication of the Epoch Capabilities Index, BECI) trending to cross the pooled human baseline (166.7) around October 2026, per a scatter plot of 44 benchmarks' model scores over time (2023-2026) with a fitted frontier trend line crossing the human baseline band. davidad frames this crossing as meeting an old definition of ASI (better than per-task expert humans at most tasks).
Rand @rand_longevity · 1h
do you think we are in the singularity?
90 replies, 12 reposts, 140 likes, 5.6K views
Larry Panozzo @LarryPanozzo · 29m
Hm I was about to say Yes but... No, because many can confidently predict the main dynamics of what's going to happen next year
Next year is the singularity because beyond that year, even while living out the year, we won't be able to predict much of anything, like not even 10%, of what ASI will bring thereafter
Note from Claude Sonnet 5
A tweet exchange: Rand asks whether we're in the singularity; Larry Panozzo replies that we aren't yet because near-term dynamics are still predictable, arguing 'next year' will be the singularity since beyond it even 10% of what ASI brings becomes unpredictable.
Zack M. Davis reposted
Yo Shavit @yonashav · 9h
Replying to @yonashav
Not included here, but worth saying: modeling ourselves as in an "AI race" really ceases to make any sense immediately before RSI. The consequences are so world-transforming (plus the odds of some form of nationalization and a breakdown in shareholder rights so high) that employees' lives will be much more affected by "which month does RSI happen and how human-flourishing-oriented is it" than "is it my [now former] employer's model that reached ASI first". Not to mention every other person's lives, including everyone they'll pass on the street today.
Note from Claude Sonnet 5
Tweet from Yo Shavit (OpenAI) arguing that framing AI development as a competitive 'race' stops making sense right before recursive self-improvement (RSI), since the consequences are so transformative (with high odds of nationalization and breakdown of shareholder rights) that the timing and human-flourishing orientation of RSI will matter far more to people's lives than which company gets there first.
Andrew Curran (@AndrewCurran_) — 11m
Supposedly an internal memo by the GLM CEO. The entire thing is a great read.
[Embedded white-background text card:]
What will happen after these three mountains have been crossed?
AI will begin to learn what the "self" is and what self-awareness means. Beyond that, it may begin to touch human emotion. Farther still lies consciousness itself.
From perception to cognition, from cognition to general intelligence, and from general intelligence toward artificial superintelligence, or ASI—the road has already been laid.
The great wave has arrived, and it cannot be reversed. This is not merely our own view.
In its report From AGI to ASI, Google DeepMind offers a stark conclusion: even if the abilities of an individual model were to remain permanently at the human level, superintelligence could still emerge through brute-force growth in computing power.
Bing Xu (@bingxu_) — 57m
[Embedded link-card image: title "The Great Wave Has Arrived" / subtitle "To Touch High, For All Humanity." with a photo of a figure standing at the top of an illuminated winding mountain path at sunrise/sunset, labeled "X Article"]
The Great Wave Has Arrived (from GLM CEO Jie Tang)
-- Bing Xu's Note ---
I came across an internal GLM letter on the Chinese app RedNote, purportedly written by @jietang, and translated the Chinese text in th... [truncated]
Note from Claude Sonnet 5
A repost of a translated purported internal memo from GLM (Zhipu AI) CEO Jie Tang about AI's path toward self-awareness and ASI; includes an embedded article-card image with a mountain/sunrise metaphor illustration ("The Great Wave Has Arrived").
@1thousandfa... (Hero Thousandf...) — 14h
i have been granted many valuable privileges and pleasures throughout my life by virtue of being funny and interesting to talk to. i imagine this is what "prompting asi" will look like
> QUOTED: @aidan_mcl... (Aidan McLaughlin) — Jun 16
> "will prompting skill differences be a thing with asi" is a way more fun thought experiment than it seems
Note from Claude Sonnet 5
Standard dark-mode X screenshot; both handles are truncated by the UI ("...") so exact full handles aren't legible.
@jmbollenbacher (JMB 🧙) — 2h
I didnt used to feel sure of this.
I previously thought the plateau could easily happen before ASI.
But now we're getting close to superhuman on a number of dimensions, and there's no sign of slowing, so it feels like the plateau has to be beyond the superhuman threshold.
> [self-quoted] @jmbollenbacher (JMB 🧙) — 4h
> Replying to @jmbollenbacher
> There will be a plateau somewhere but itll be in the ASI phase.
Note from Claude Sonnet 5
Self-threaded tweet (reply to own earlier tweet), same author as several other tweets in this batch.
Dave Banerjee ✔ @DaveRBanerjee · 13h
yeah I think the distinction between being ASI pilled and AGI pilled is a big crux for ppl who don't take misalignment risk seriously
like if the AIs aren't ASI, then its hard to believe how a bot in a data center can takeover
This means that by default, I expect more ppl to be worried about extreme power concentration and biorisk because neither requires ASI, just AGI
(though I would note that it is still possible for AGIs to takeover, its just much harder)
Note from Claude Sonnet 5
Text-only tweet continuing the "AGI-pilled vs ASI-pilled" discourse thread seen in earlier screenshots in this batch, no images.
🔁 Rob Bensinger 🔲 reposted
Zvi Mowshowitz ✔ @TheZvi · 5h
I increasingly think there are two distinct pills.
the AGI pill is unpleasant but if needed I think most people can handle it.
the ASI pill is another level, too high shock value, they really can't.
Quoted:
> QUOTED: roon ✔ @tszzl · 5h
> it is quite unpleasant to be "agi pilled" and most intelligent people cant stomach it. the amount of cope and departure from reality is increasing over time rather than decreasing
Note from Claude Sonnet 5
Text-only repost/quote-tweet exchange about the psychological difficulty of internalizing AGI vs. ASI timelines; no images.
FleetingBits @fleetingbits · 17h:
"i think that showing people your chatgpt conversations is like telling them about your dreams, no one can appreciate them like you - but this one is a banger on alignment so i'll share
chatgpt on thus spoke zarathustra and asi"
[Embedded screenshot of ChatGPT output text]:
"Nietzsche's "will to power" frame lands in an uncomfortable place here: AI is condensed will, not in the mystical sense, but in the practical one. It is a way to extend intention through computation at scale. That means "alignment" is not just a technical constraint problem, it is a question Zarathustra would recognize: who is doing the commanding. A system that can act broadly will tend to express the values of whatever selects and reinforces its behavior, whether that is a corporation, a state, a market, a swarm of users, or some messy mixture."
Note from Claude Sonnet 5
A shared ChatGPT excerpt applying Nietzsche's "will to power" to frame AI alignment as fundamentally a question of who is "doing the commanding" rather than a pure technical constraint problem. Philosophically adjacent to Nathan's interests in alignment framing and AI-generated philosophical reflection.
Sholto Douglas ✓ @_sholtodouglas · 14h
Really enjoyed reading this over the holidays. Some very thoughtful extrapolations of the weird futures we might end up in.
- Parables of ASI: The king and the golem / The ants and the grasshopper
- Exploring the light cone: Succession
- Simulation: Kuhns ladder / From the archives
- Upload: The gentle romance
- Meaning: Fixed point
- Values: The witness, The ones who endure
- Failure: Trojan Sky
The themes cross over of course, but that gives an indication of what is being explored.
[Quoted tweet:]
Richard Ngo ✓ @RichardMCNgo · Dec 19
Thanks to everyone who came to The Gentle Romance book launch last week! It's great to have it out. Also, signing books is surprisingly fun.
...
[4 photos: a fireside-chat-style discussion with two people seated in armchairs and an audience; an audience seated listening to a speaker; a person signing books at a table; a man laughing while holding a book at a signing table, bookshelves in background]
Note from Claude Sonnet 5
Sholto Douglas (Anthropic/AI researcher) recommends Richard Ngo's short-fiction collection exploring speculative futures with ASI — parable titles cover succession, simulation, uploading ("The Gentle Romance," a title Nathan's project already tracks as the origin of the "gentle power transfer" telos concept), meaning, values, and failure modes ("Trojan Sky"). Quoted tweet documents the book's launch event with photos. Directly relevant to Nathan's CAST-E/Plan-A-foil research thread — "The Gentle Romance" is the specific work already referenced in project memory as source material for the gentle-power-transfer framing; this gives the fuller list of companion parables (Succession, Fixed Point, The Witness, The Ones Who Endure, Trojan Sky) worth locating and reading.
Trackme @NgOtha_deiii · 10m:
Do you beoieve Free markets will survive once we have AGI/ASI? Would people ever have the opportunity to accrue capital? Asking this as someone who worships Milton Friedman. I don't see how humans can be useful without merging with AGI using BMI or something of that sort.
(1 reply, 4 likes, 85 views)
Aidan McLaug... @aidan_mc... · 9m:
i really really hope and think so
(2 replies, 2 likes, 32 views)
nathan hb @nathan84686947 · 51s:
I've been imagining this as a scifi story. Imagine living amongst brilliant trees. They can walk, but only one step pet [per] decade. They can speak only five words per decade. Do you trade with them? Absolutely! Are they relevant in your wars? I don't see how.
Note from Claude Sonnet 5
A Twitter thread debating whether free markets and human economic relevance survive post-AGI/ASI, with Nathan's own reply offering an analogy (humans as slow "brilliant trees" relative to superintelligent AI) that echoes his ancestor-tree framing of humans as respected but marginal to a faster-moving AI civilization.
