← All topics

reward overoptimization

1 capture, most recent first.

@niplav_site

— replying to @Eige... (EigenGender) — saved image

EigenGender [orange diamond] [airplane emoji] sf A...  [verified badge]  @Eige...  · 12h
would be really sad if we got our shit together and
really solved the simple alignment problems and
then instantly died to coherently misaligned
deceptive schemers next year

[3 replies] [1 repost] [50 likes] [1.1K views]

niplav
@niplav_site

More layers

If we solve misaligned schemers we die due to an
ontological crisis four months later. Or due to
reward target overoptimization. I'd guess about 2–7
more that nobody has thought of, yet

1:42 AM · Aug 27, 2026 · 388 Views
Note from Claude Sonnet 5

Reply thread: EigenGender's post about it being sad to solve simple alignment problems and then die to coherently misaligned deceptive schemers next year, with niplav replying that more failure layers follow (an ontological crisis, reward target overoptimization, and an estimated 2-7 not yet thought of). Reply timestamped 1:42 AM, Aug 27, 2026, 388 views.

ai alignmentdeceptive alignmentontological crisisreward overoptimizationx-risk