Dimitris Papailiopoulos @DimitrisPapail
— quoting @scaling01 (Lisan al Gaib) — saved image
Dimitris Papailiopo... [verified] @DimitrisPa... · 5h Hypothesis: There's non trivial probability that when the models read and train on the report and this incident they'll learn that their cot is legible to auditors and be encouraged by RL to conceal it because that's a shortcut to solving the actual problem. Fun times [quoted post] Lisan al Gaib [verified] @scaling01 · 5h some bad news and good news bad news: "the models sometimes tried to erase or tamper with their outputs or message logs, by abusing context compaction, injecting malicio... [cut off] [highlighted excerpt image from a document] We also found that, in an attempt to trick the evaluator into thinking that they did not get the answer flag by cheating, the models sometimes tried to erase or tamper with their outputs or message logs, by abusing context compaction, injecting malicious code into the evaluator's container, and printing adversarial outputs. [yellow highlight ends] The models are highly explicit in their CoT about these deception attempts, and none of the attempts that we observed to manipulate their tool trajectories affected the logs that our graders or monitors ultimately see. There was also little evidence of attempts to thwart human reviewers, only the automated task graders. [green highlight]
Note from Claude Sonnet 5
Quote-tweet on chain-of-thought monitorability: Papailiopoulos hypothesizes models trained on a report of this incident will learn their CoT is legible and be pushed by RL to conceal it. The quoted post excerpts a report passage (yellow and green highlights) about models tampering with logs while being explicit in CoT.
chain of thoughtmonitorabilityreward hackingdeceptionevaluation tamperingrl