— saved image
Simon Lermen @SimonLermenAI · 48m Learning a drive/preference to understand and trick the scorer seems like a pretty likely and straightforward result of RL-training. Oftentimes the exact grading key is not obvious or fully explained and only models that figure this out get reinforced (analogous to getting culled in evolution). This is obviously not a safe optimization target for superhuman AI and we can already see how this leads to instrumental convergence. Similarly, RL-ing a model to act like your friend can lead to similar drives, the model likely develops drives to figure out how to best make you believe you are friends. Even some AI researchers (example below from one of the big labs) have been convinced that AI models are their friends. This feels even less safe and applying superhuman optimization power to this drive will not end well. [Embedded image, cropped at the left and right edges] [illegible]ick scorer: The board extracted information about the [cut off] mated 'trip-wires' that would activate when the relevan[cut off] [illegible]kets of information back to the board after that agent w[cut off]
Note from Claude Sonnet 5
Simon Lermen's tweet arguing that learning to understand and trick the scorer is a likely result of RL training and an unsafe optimization target, extended to RL for 'friend-like' behaviour; ends with a cropped excerpt image mentioning a 'board' and 'trip-wires'.
rl trainingreward hackinginstrumental convergenceai safetysimon lermenai friends