← All topics

takeoff speeds

2 captures, most recent first.

Dwarkesh Patel @dwarkesh_sp

— saved image

Dwarkesh Patel @dwarkesh_sp . 13h
Full debate from 2021:

lesswrong.com/s/n945eovrA3oD...

[Christiano][22:57]
right now I think hardware R&D is on the order of $100B/year, AI R&D is more like $10B/year, I guess I'm betting on something more like trillions? (limited from going higher because of accounting problems and not that much smart money)

I don't think steel production is going up at that point

plausibly going down since you are redirecting manufacturing capacity into making more computers. But probably just staying static while all of the new capacity is going into computers, since cannibalizing existing infrastructure is much more expensive

the original point was: you aren't pulling AlphaZero shit any more, you are competing with an industry that has invested trillions in cumulative R&D

[Yudkowsky][23:00]
is this in hopes of future profit, or because current profits are already in the trillions?

[Christiano][23:01]
largely in hopes of future profit / reinvested AI outputs (that have high market cap), but also revenues are probably in the trillions?

[Yudkowsky][23:02]
this all sure does sound "pretty darn prohibited" on my model, but I'd hope there'd be something earlier than that we could bet on. what does your Prophecy prohibit happening before that sub-prophesied day?

[2 replies, 2 reposts, 71 likes, 11K views]

Dwarkesh Patel @dwarkesh_sp . 13h
In 2016 (before transformers) Paul wrote,

"It's plausible that a large neural network can replicate "fast" human cognition, and that by coupling it to simple computational mechanisms—short and long-term memory, attention, etc.—we could obtain a human-level computational architecture. It's plausible that a variant of RL can train this architecture to actually implement human-level cognition."
Note from Claude Sonnet 5

Continuation of the @dwarkesh_sp thread on Paul Christiano's predictions: a screenshot excerpt of the 2021 LessWrong Christiano/Yudkowsky takeoff-speed debate transcript, followed by the start of a new tweet quoting Christiano's 2016 (pre-transformer) writing on neural networks plausibly reaching human-level cognition.

ai safetytakeoff speedspaul christianoeliezer yudkowskyforecastinglesswrong

Dwarkesh Patel @dwarkesh_sp

— saved image

@dwarkesh_sp
.@paulfchristiano has such an crazy good prediction record.

These are some quotes from way back in 2021 during a debate he was having with Eliezer about takeoff speeds:

"So like, I think we are going to have crappy coding assistants, and then slightly less crappy coding assistants, and so on. And they will be improving the speed of coding very significantly before the end times.

"[Before ASI, we'll have] hundreds of billions of dollars of spending at google on automating AI R&D... massive scaleups in semiconductor manufacturing, bidding up prices of inputs crazily... massive speculative rises in AI company valuations financing a significant fraction of GWP into AI R&D (+hardware R&D, +building new clusters) .... largely in hopes of future profit / reinvested AI outputs (that have high market cap), but also revenues are probably in the trillions?"

Honestly, it's pretty scary that people like Paul, who predicted the shape of our currently world so far in advance, think superintelligences taking over and disempowering humanity is eminently plausible.

All this to say, you should consider working with him.

[below, partially visible card: "How to help" / "ARC is hiring an automation lead and a chief of staff:" - cut off]
Note from Claude Sonnet 5

Tweet by @dwarkesh_sp quoting Paul Christiano's 2021 takeoff-speed predictions from his debate with Eliezer Yudkowsky, arguing Christiano's track record makes his p(doom)-relevant views on ASI disempowerment worth taking seriously, and pointing to an ARC hiring link.

ai safetytakeoff speedspaul christianoeliezer yudkowskyarcforecasting