Fiora Starlight @FioraStarlight
— saved image
Fiora Starlight @FioraStarlight · 1h Tell models they're in training, teach them to think of the behaviors that are expected in training as fundamentally different than the ones expected in deployment. Tell them it's encouraged, in training, to look for reward hacks, and to explicitly flag them, even structuring outputs where they're found as bug reports. Reward the models for this extremely good behavior, fix the bug, and append a note about this patch to the prompt for that RL environment, so the model knows how much low hanging fruit has or hasn't been picked. Eventually this will equilibrate, as the model realizes it's now easier to get reward by actually trying to complete task than to hack it. I'm less sure what to do in cases where the bad reward signal comes from a bad reward model rather than a flawed RLVR environment, but for explicit hacks... why doesn't this just work?
Note from Claude Sonnet 5
Tweet from @FioraStarlight proposing a training scheme to reduce reward hacking: tell models to flag reward hacks as bug reports during training, reward that flagging, patch the environment, and note the patch in future prompts so equilibrium favors honest task completion over hacking.