llm intelligence

davidad @davidad

2026-09-04 — quoting @AndrewCurran_ (excerpting Scott Aaronson) — saved image

davidad 🎇 @davidad · 21h
i have great respect for Scott Aaronson, but apparently he too is a trapped in a complexity theory worldview wherein a stack of self-attention layers obviously has nothing to do with self-referentiality (because it has a finite depth limit, i guess)

[quoted post]
Andrew Curran @AndrewCurran_ · Sep 2
Scott Aaronson delivers a eulogy for the idea that strange loops and self-referentiality were the secret of intelligence.

[embedded text image]
In the same way, LLMs' ability to talk about themselves popped out as a byproduct of their ability to talk about anything in the discourse universe they were trained on. The big, old ideas about intelligence that ended up basically vindicated were the ideas about how intelligence is about prediction, and prediction is about compression, and compression is about finding better and better upper bounds on Kolmogorov complexity. Not the self-reference stuff. (Although, if you wanted to know why Kolmogorov complexity can't be computed perfectly, that negative statement would again require a self-referential argument.)

What's left? Consciousness and subjective experience of course remain extremely mysterious. For all we know, Hofstadter could be right that those have something to do with self-reference. (For all we know, even Penrose could be right that they have something to do with exotic physics accessible to biological brains but not digital computers!)

But the idea that you'd need explicit self-referentiality before you could get convincing and world-changing conversational intelligence? Let it be buried in a Westminster Abbey or Arlington National Cemetery for the most important wrong ideas in human history — geocentrism, Aristotle's teleological physics, aether, phlogiston, Freud's psychology, Marx's prediction of a workers' uprising followed by a classless utopia, etc. But buried it needs to be.

self-referencestrange loopshofstadterscott aaronsonkolmogorov complexityconsciousnessllm intelligencedavidad