Binfeng Xu

Binfeng Xu @billxbf · May 29 Besides token faithfulness (TITO), there are a few more challenges I noted in long form agent RL, tldr: - Rollout takes 80%+ overall time. Long tail (eg. looping errors) rollout are ubiquitous, and so efficient async RL is a must. - Correctly handling policy drifting during async RL. Strike a balance between efficiency and correctness (staleness). - Scarcity of reward is a pain. Simple outcome testing can encourage suboptimal intermediate steps. So PRM style correction (by the right amount) is important. - Environment cleanness and consistency are crucial. Reward hacking usually results from dirty env construction (eg. leaking files). Besides, mismatch between training environments and test-time harness harms more than you think. - I wrote more about these in a recent blog here: billxbf.github.io/posts/agent-rl... we solved most these problem with Polar and are patching up the rest. Stay tuned for upcoming updates! [Quoted tweet:] > QUOTED: clem 🤗 @ClementDelangue · May 28 > Most people training agentic LLMs with RL right now have a silently broken training loop and have no idea. > Here's the trap: single-turn RL works beautifull... > [Embedded diagram: "Multi-turn RL loop, with a tool call." — sample prompt [{"role": "user", "content": "What's 2+2?"}] → tokenize prompt [10 23 42 17 99] "<user>What's 2+2?</user><eos>" → generate completion [50 71 13 99] "<tool_call>calc(2+2)</tool_call><eos>" → execute tool and append result [60 4 61 99] "<result>4</result><eos>" → generate completion [4 7 99] "4.<eos>" → compute reward +1 → backprop on assistant [50 71 13 99] + [4 7 99]]