Wei Dai @weidai11
— quoting @weidai11 — saved image
Steve Bachelor reposted Wei Dai @weidai11 · 14h We can't easily evaluate long-horizon strategy, even after the horizon has passed and we can judge in hindsight, because we don't know what the counterfactual world looks like. Maybe the only way to get credit for many strategy ideas is to wait for large-scale high-fidelity simulation abilities, post-Singularity. But perhaps that's still a better situation than with philosophy, where even post-Singularity, there's no guarantee that superintelligences will converge to correct philosophical conclusions. They may inherit or develop wrong philosophical traditions (methods of reasoning), become stuck with wrong ideas (e.g., about the nature of consciousness or normativity), and still take over the universe. [quoted post] Wei Dai @weidai11 · Mar 26 One idea is that humans are "aligned" via social status, but this causes an "alignment tax" forcing the smartest humans to work in domains where their results can be evaluated by others less intelligent.... [cut off]
Note from Claude Sonnet 5
Wei Dai post arguing that long-horizon strategy cannot be evaluated even in hindsight (no counterfactual), and that philosophy is worse off since superintelligences might converge on wrong philosophical conclusions about consciousness or normativity yet still take over; quotes his earlier post on humans being 'aligned' via social status.
metaphilosophylong-horizon strategysuperintelligenceconsciousnessnormativityalignmentwei dai