— saved image
roon reposted Tom McGrath (verified) @banburismus_ · 6h probably a very obvious point but reward specification is a fundamentally unsolvable problem. you're asking me to specify how much I like all possible states but: 1. I can't imagine all possible states 2. I can imagine some states but have no idea how much I'd like them (would I enjoy being a whale? idk) 3. I can imagine some states, am confident in how much I'd like them, but am wrong
Note from Claude Sonnet 5
Tweet arguing reward specification is fundamentally unsolvable, because one cannot imagine all states, cannot predict liking for some, and is wrong about others.