— web clipping, 445 words — published 2026-10-05
Post by @herbiebradley on X
Currently I still only know of 1 alignment-specific RL environment company: making RL envs that aim to represent situations in real deployments in which alignment is challenging.
It seems like a huge mistake in hindsight that we have seen so much encouragement and funding to, for example, start 3rd party evaluators, but not comparable encouragement to start alignment data companies.
I do think a lot of traditional AI safety views underestimate the importance of data, so perhaps this is related. But the Hugging Face incident seems squarely downstream of both (a) reward hackable RL envs in general (b) much less effort and $ put into RL envs where alignment is a core component of reward.
A separate idea is alignment by default: the old idea of "find the persona basin which is aligned" and things will just work. This is clearly not guaranteed under increased RL. But my sense is there has not been a huge amount of effort to figure out how custom environments could help "nudge" the model back into good persona basins during extensive RL. Here too, a robust ecosystem of alignment data companies could help significantly.
In the RL data market, there is a trend for some labs to in-house slightly more RL env production. But I don't expect this to fully disappear, and I am convinced the market-based approach to sourcing RL envs produces the best overall volume and quality—for alignment RL, an ecosystem of startups would also greatly help make behind-frontier models safer.
---
##### Comments
> **FleetingBits @fleetingbits** · [2026-10-05](https://x.com/fleetingbits/status/2107170217394905195)
>
> banger
> **sensho @sensho** · [2026-10-05](https://x.com/sensho/status/2107188844915482724)
>
> we do this or dont do this depending on your definition of real deployments. i agree the industry is going to need many more beyond us or the other co
>
> we plan on open sourcing frameworks and envs for this kind of work so hopefully that will encourage more and/or make it easier to start
> **Fajar M Reza @FReza1984** · [2026-10-05](https://x.com/FReza1984/status/2107178512172171457)
>
> Deployment-shaped RL environments could expose alignment failures before expensive production incidents.
> **Connacher Murphy @connacher\_** · [2026-10-05](https://x.com/connacher_/status/2107181055660671427)
>
> Trying to do this via a marketplace and I always welcome feedback!
> **toni @tkukurin** · [2026-10-05](https://x.com/tkukurin/status/2107174727953338460)
>
> RL env companies should both better audit their evals and properly invest in eval divisions.
>
> if you read the anthropic report it seems likely they overfit to a particular set of workflows.
> **Joshua Levy @josh\_levy\_** · [2026-10-05](https://x.com/josh_levy_/status/2107176671526932882)
>
> Presumably if third party evaluators have teeth then this will exert pressure on labs to develop or procure solutions like these.
>
> Getting evaluation right seems more important as it's upstream.