self-play

León @LeonGuertler · 1h 3/7 Code Example Using unstable baselines is relatively simple. Just import unstable (has to be installed via "pip install unstable-rl"), initialize the necessary objects and start training. To make training more accessible, beyond the computational (and vRAM) requirement reductions achieved by using LoRA, we implemented optional full activation check-pointing and training sequence truncation (i.e. only training on the first "N" tokens, but generating (and evaluating) sequences of up to "M" tokens; this works very well in practice). When using both of these, it is possible to train up to 4B parameter models using self-play on just 24gb of vRAM (although it is worth mentioning that currently you will need at least 2 GPUs (i.e. minimum of 2x24gb) since learners and actors (currently) can't share the same gpu). This is also a good time to explain why we built this library "LoRA first". UnstableBaselines is supposed to be a hackable/fast prototyping library making it easy to not just iterate on ideas, but also research topics that would be harder to research w/o lora (i.e. opponent sampling). Furthermore, as shown by RAGEN/Tina/etc. reasoning (at least seemingly) can sufficiently be trained via low-rank updates. [Partially visible code snippet below:] import ray, unstable import unstable.reward_transformations as retra tracker = unstable.Tracker.options(name="Tracker").remote(run_name="demo", wandb_project="UB") step_buffer = unstable.StepBuffer.options(name="StepBuffer").remote(...