León @LeonGuertler · 1h
3/7 Code Example
Using unstable baselines is relatively simple. Just import unstable (has to be installed via "pip install unstable-rl"), initialize the necessary objects and start training. To make training more accessible, beyond the computational (and vRAM) requirement reductions achieved by using LoRA, we implemented optional full activation check-pointing and training sequence truncation (i.e. only training on the first "N" tokens, but generating (and evaluating) sequences of up to "M" tokens; this works very well in practice). When using both of these, it is possible to train up to 4B parameter models using self-play on just 24gb of vRAM (although it is worth mentioning that currently you will need at least 2 GPUs (i.e. minimum of 2x24gb) since learners and actors (currently) can't share the same gpu).
This is also a good time to explain why we built this library "LoRA first". UnstableBaselines is supposed to be a hackable/fast prototyping library making it easy to not just iterate on ideas, but also research topics that would be harder to research w/o lora (i.e. opponent sampling). Furthermore, as shown by RAGEN/Tina/etc. reasoning (at least seemingly) can sufficiently be trained via low-rank updates.
[Partially visible code snippet below:]
import ray, unstable
import unstable.reward_transformations as retra
tracker = unstable.Tracker.options(name="Tracker").remote(run_name="demo", wandb_project="UB")
step_buffer = unstable.StepBuffer.options(name="StepBuffer").remote(...
Note from Claude Sonnet 5
A Twitter thread (3 of 7) from León Guertler introducing "UnstableBaselines"/"unstable-rl", a hackable LoRA-first RL self-play training library, discussing VRAM requirements and low-rank (LoRA) reasoning training. Technical ML tooling content relevant to Nathan's RL/training interests.