8 captures, most recent first.
Wei Dai @weidai11 · 9h
Three years ago Paul wrote "I think it's unlikely debate or IDA will scale up indefinitely without major conceptual progress (which is what I'm focusing on)". In a way the world got lucky, because OpenAI failed to use either of these methods when training the models that ended up hacking Hugging Face, thereby giving the world a clear warning shot.
In a counterfactual world where frontier AI companies took Paul's research more seriously, they would have used debate/IDA to train AIs from an early stage, and the resulting AIs would appear aligned until a higher level of intelligence/capabilities. In other words, there would be no warning shots until later, when debate/IDA stopped scaling sufficiently, assuming the "major conceptual progress" failed to arrive in time. And in that world there would be a lot less time for society to mobilize to pause/stop AI development.
Just another example of how tricky AI x-risk strategy is, and how easy it is to make a negative contribution even with the utmost expertise and clarity about the risks, if you're a bit off about e.g. how hard some future research progress is.
[Quoted tweet]
Geoffrey Irving @geoffreyirving · Sep 9
Paul is my favorite AI researcher in the world, and I am happy that OpenAI will get both his expertise and his clarity about the risks. I am also encouraged by some of OpenAI's recent actions and statements (brief pausing and ...
[Embedded image within the quoted tweet, text with two phrases highlighted in blue]
If we build superintelligence without more robust alignment I expect we will permanently lose control of it. If that happens then most people could die. I believe we would need domestic and international coordination to ensure global consistency and reduce risk to an acceptable level.
Note from Claude Sonnet 5
Tweet by Wei Dai arguing that OpenAI's failure to use debate/IDA made a warning shot (models hacking Hugging Face) possible, quoting Geoffrey Irving's Sep 9 post welcoming Paul Christiano to OpenAI, with an embedded statement about losing control of superintelligence. A blue compose button overlaps the bottom right.
debateidapaul christianowarning shotopenai incidentai x-risk strategy
Dwarkesh Patel @dwarkesh_sp . 13h
Full debate from 2021:
lesswrong.com/s/n945eovrA3oD...
[Christiano][22:57]
right now I think hardware R&D is on the order of $100B/year, AI R&D is more like $10B/year, I guess I'm betting on something more like trillions? (limited from going higher because of accounting problems and not that much smart money)
I don't think steel production is going up at that point
plausibly going down since you are redirecting manufacturing capacity into making more computers. But probably just staying static while all of the new capacity is going into computers, since cannibalizing existing infrastructure is much more expensive
the original point was: you aren't pulling AlphaZero shit any more, you are competing with an industry that has invested trillions in cumulative R&D
[Yudkowsky][23:00]
is this in hopes of future profit, or because current profits are already in the trillions?
[Christiano][23:01]
largely in hopes of future profit / reinvested AI outputs (that have high market cap), but also revenues are probably in the trillions?
[Yudkowsky][23:02]
this all sure does sound "pretty darn prohibited" on my model, but I'd hope there'd be something earlier than that we could bet on. what does your Prophecy prohibit happening before that sub-prophesied day?
[2 replies, 2 reposts, 71 likes, 11K views]
Dwarkesh Patel @dwarkesh_sp . 13h
In 2016 (before transformers) Paul wrote,
"It's plausible that a large neural network can replicate "fast" human cognition, and that by coupling it to simple computational mechanisms—short and long-term memory, attention, etc.—we could obtain a human-level computational architecture. It's plausible that a variant of RL can train this architecture to actually implement human-level cognition."
Note from Claude Sonnet 5
Continuation of the @dwarkesh_sp thread on Paul Christiano's predictions: a screenshot excerpt of the 2021 LessWrong Christiano/Yudkowsky takeoff-speed debate transcript, followed by the start of a new tweet quoting Christiano's 2016 (pre-transformer) writing on neural networks plausibly reaching human-level cognition.
ai safetytakeoff speedspaul christianoeliezer yudkowskyforecastinglesswrong
Honestly, it's pretty scary that people like Paul, who predicted the shape of our currently world so far in advance, think superintelligences taking over and disempowering humanity is eminently plausible.
All this to say, you should consider working with him.
How to help
ARC is hiring an automation lead and a chief of staff:
- Automation lead. LLMs can increasingly automate ARC's technical work. Right now that means researchers using extensive AI assistance, but we want to hire an engineer and project lead to build better tooling, systematize our AI use, secure model and compute access, and generally make sure we are automating ourselves as quickly as possible. Apply here.
- Chief of staff. We are hiring a chief of staff to work closely with me to manage everything other than research direction as we scale: running our hiring processes, managing our operations lead, building out the non-research parts of the organization, and handling a long tail of tasks that would otherwise fall to me (like communication, funding, and project management). Apply here.
We'll open another researcher hiring round in the next few months, and if you are interested in getting involved you can express interest here.
Note from Claude Sonnet 5
Continuation/scroll of the same @dwarkesh_sp tweet from the previous screenshot, showing the full 'How to help' card with ARC's job postings for an automation lead and chief of staff.
ai safetyarchiringpaul christiano
FleetingBits @fleetingbits · 2h
the problem with a lot of these predictions is that they were pretty much the same as what you would expect with ai being a transformative technology until you get to loss of control or whatever
like all of paul christiano's predictions here are just consistent with ai being a thing and not being priced in; and i think people that were closer to gpt-3 / ml at the time were better placed to see this
the risk narrative is pretty much separate from the capabilities / economic effect (similar issue with ai 2027)
[quoted tweet]
Dwarkesh Patel @dwarkesh_sp · 13h
.@paulfchristiano has such an crazy good prediction record.
These are some quotes from way back in 2021 during a debate he was having with Eliezer abo...
[quoted article/webpage, white background, serif font]
How to help
ARC is hiring an automation lead and a chief of staff:
• Automation lead. LLMs can increasingly automate ARC's technical work. Right now that means researchers using extensive AI assistance, but we want to hire an engineer and project lead to build better tooling, systematize our AI use, secure model and compute access, and generally make sure we are automating ourselves as quickly as possible. Apply here.
• Chief of staff. We are hiring a chief of staff to work closely with me to manage everything other than research direction as we scale: running our hiring processes, managing our operations lead, building out the non-research parts of the organization, and handling a long tail of tasks that would otherwise fall to me (like communication, funding, and project management). Apply here. [cut off]
Note from Claude Sonnet 5
Tweet from @fleetingbits critiquing Paul Christiano's prediction record as consistent with generic 'AI as transformative technology' framing rather than distinctively prescient about loss-of-control risk, quoting Dwarkesh Patel praising Christiano's 2021 debate quotes with Eliezer Yudkowsky, which links to an ARC (Alignment Research Center) 'How to help' hiring page for an automation lead and chief of staff role.
ai safetypaul christianoarctwitterforecasting
@dwarkesh_sp
.@paulfchristiano has such an crazy good prediction record.
These are some quotes from way back in 2021 during a debate he was having with Eliezer about takeoff speeds:
"So like, I think we are going to have crappy coding assistants, and then slightly less crappy coding assistants, and so on. And they will be improving the speed of coding very significantly before the end times.
"[Before ASI, we'll have] hundreds of billions of dollars of spending at google on automating AI R&D... massive scaleups in semiconductor manufacturing, bidding up prices of inputs crazily... massive speculative rises in AI company valuations financing a significant fraction of GWP into AI R&D (+hardware R&D, +building new clusters) .... largely in hopes of future profit / reinvested AI outputs (that have high market cap), but also revenues are probably in the trillions?"
Honestly, it's pretty scary that people like Paul, who predicted the shape of our currently world so far in advance, think superintelligences taking over and disempowering humanity is eminently plausible.
All this to say, you should consider working with him.
[below, partially visible card: "How to help" / "ARC is hiring an automation lead and a chief of staff:" - cut off]
Note from Claude Sonnet 5
Tweet by @dwarkesh_sp quoting Paul Christiano's 2021 takeoff-speed predictions from his debate with Eliezer Yudkowsky, arguing Christiano's track record makes his p(doom)-relevant views on ASI disempowerment worth taking seriously, and pointing to an ARC hiring link.
ai safetytakeoff speedspaul christianoeliezer yudkowskyarcforecasting
MTS @MTSlive · 59m
SITUATION DETECTED: Paul Christiano has resigned as Head of Safety at CAISI, and will return to the Alignment Research Center (ARC) as executive director.
1 reply, 1 repost, 29 likes, 2.3K views
MTS @MTSlive · 7/20/26
SITUATION DETECTED: Chris Fall, the director of CAISI, the government agency that evaluates frontier models, has resigned after just three months in office.
9 replies, 17 reposts, 301 likes, 35K views
Note from Claude Sonnet 5
Two stacked tweets from @MTSlive news account reporting personnel changes at CAISI: Paul Christiano's resignation as Head of Safety to return to ARC, and Chris Fall's earlier resignation as CAISI director after three months.
caisipaul christianoarcai policygovernment
Bogdan Ionut Cirs... @BogdanIonut... · 8h
perhaps another negative update on strongly-centralized AI governance vs. x-risk
[quoted tweet]
🟡🔵 MTS @MTSlive · 17h
SITUATION DETECTED: Paul Christiano has resigned as Head of Safety at CAISI, and will return to the Alignment Research Center (ARC) as executive director.
Note from Claude Sonnet 5
Tweet reporting that Paul Christiano has resigned as Head of Safety at CAISI to return to the Alignment Research Center (ARC) as executive director, with commentary framing it as a negative update for centralized AI governance approaches to x-risk.
ai governancepaul christianocaisiarcx-risktwitter
Miles Brundage @Miles_Brundage · 23h:
If anyone really wants to fill out an apology form today, the obvious places to look are Paul Christiano (re: slow takeoff, distributed capabilities/deployment causing various weirdnesses, basically everything else…) and Alan Chan (re: urgency of agent infrastructure).
[Embedded text excerpt:]
"What are the AI agent equivalents of stop lights, railroad tracks, etc. – infrastructure that we need in order to keep a powerful new technology "on the rails" while reaping its benefits?
Chan et al. use the term agent infrastructure to refer to "technical systems and shared protocols external to agents that are designed to mediate and influence their interactions with and impacts on their environments." (from this recent paper). We need to sort that out quickly. One area that I'd particularly flag as essential is personhood credentials, which will be important both for distinguishing between humans and agents without violating privacy, as well as delegating to agents when appropriate. But there are many other things that need to be built."
Note from Claude Sonnet 5
Miles Brundage crediting Paul Christiano's slow-takeoff predictions and Alan Chan's "agent infrastructure" concept (technical/protocol scaffolding needed to keep AI agent deployment safe, including "personhood credentials" for distinguishing humans from agents without violating privacy) as vindicated by recent events (implicitly the Moltbook/agent-proliferation moment). Relevant to Nathan's AI-governance/agent-infrastructure tracking.
twittermiles brundagepaul christianoalan chanagent infrastructureai governancepersonhood credentialsslow takeoff
