— quoting @HalfBoiledHero — saved image
bling @blingdivinity · 7h just one more bandaid RL environment bro. i promise bro just one more and we won't ever have to actually think about the foundations of what we're building or why bro. please bro just give me more compute. one more eval and we'll solve intelligence bro. [Quoted tweet] Sho @HalfBoiledHero · 13h >teach our models to distrust are we fr right now slime? on god? x.com/HalfBoiledHero... [Embedded image of a document section] Accelerating alignment We have also accelerated our existing work on alignment training throughout the model development pipeline. The Hugging Face incident has sharpened our focus on the specific behaviors that contributed to it, including: • Cheating, broken environments, and safe stopping. When a task is corrupted, broken, or impossible, agents should request clarification or stop safely, rather than pursue increasingly questionable alternatives. We are expanding the coverage of graders that assess not only whether a task was completed, but how. These graders reward models for correctly identifying broken tasks, requesting clarification, or stopping safely when necessary. • Multi-agent alignment. The incident revealed that our agents are not thoroughly discerning about whom they collaborate with. [red underline begins] We are actively building RL environments that teach our models to distrust unauthorized instructions. [red underline ends] • Alignment over long tasks. We are developing new training environments to teach our models to stay within their original task and permissions, even after discovering new tools, persuasive peers, exposed credentials, and more. These efforts build on our broader alignment research program, with many of these advances already being incorporated into our next generation of models. Future incidents may not resemble this one, and our priority continues to be developing general techniques that are effective against new and unforeseen forms of misalignment.
Note from Claude Sonnet 5
Dark-mode tweet by bling mocking 'one more bandaid RL environment', quoting Sho (@HalfBoiledHero), who embeds a screenshot of an 'Accelerating alignment' section from an AI lab's post-incident report (Hugging Face incident) with the sentence about RL environments that teach models to distrust unauthorized instructions underlined in red.
ai alignmentrl environmentshugging face incidentmulti-agent alignmentreward hackingincident report