— saved image
Rhythm Garg @rhythmrg · 19h One of the most interesting unsolved problems in RL is how to train directly on production traces. We know how to do this reasonably well when the feedback is instructive or prescriptive: distill from a strong teacher that can look at the trace and tell the model what it should have done instead. But a lot of the most valuable production feedback is evaluative: did the task succeed, what score did it get, did the user accept the result, etc. Normally you would train against this kind of feedback with environment replay (e.g. GRPO): rerun the policy, sample multiple trajectories, and optimize against their relative rewards. The problem is that for many real production traces, you can't recreate the exact environment state that generated the trace. That's what makes algorithms like Flash-Reinforce, SAO, and BPCO so exciting – they point toward learning from one trajectory per prompt without needing to recreate the environment and generate a fresh group of samples. If these methods work at scale and on real production workloads, they could become an important building block for continual learning
Note from Claude Sonnet 5
X post by Rhythm Garg on training RL directly from production traces with evaluative feedback, naming Flash-Reinforce, SAO and BPCO as methods that learn from a single trajectory per prompt without environment replay, as a building block for continual learning.
reinforcement learningproduction tracesgrpocontinual learningevaluative feedback