Samuel Ratnam @eterecursion · 29m
sidenote but averaging weights in ML works because useful circuits constructively interfere and task specific noise destructively interferes leading to better generalisation
[Quoted tweet]
Samuel Ratnam @eterecursion · 1h
this paper is so cool: arxiv.org/abs/2410.01463
you can get the benefits of data aggregation and personalisation without compromising on your privacy. You average everyone's A matrix to lear...
[Attached diagram, labeled 'FedSA-LoRA (Ours)': shows h = (arrows from) Pretrained weights W ∈ R^(d×d) [frozen, blue] plus B=0 [pink triangle] over A = N(0, σ²) [orange trapezoid, low-rank r] combined, with input x of dimension d at bottom. Legend: blue = frozen, orange = trainable - global, pink = trainable - local.]
Note from Claude Sonnet 5
Tweet by Samuel Ratnam explaining why weight averaging works in ML (constructive interference of useful circuits, destructive interference of task-specific noise), quoting his own earlier post about the FedSA-LoRA paper (arXiv:2410.01463) on federated personalization via averaging the LoRA A matrix while keeping B local, with the paper's architecture diagram attached.
machine learningfederated learningloraweight averagingtwitter
kalomaze (@kalomaze) — 7h
getting pissed off that this:
a. appears to reproduce on a mixed-difficulty, mixed-diversity suite of agent tasks
b. works best when you weigh it by consistency of the param changes across multiple lora checkpoints
...it feels like trollface physics? wtf is the catch?
[Embedded table, dark background, columns: tasks, n/task, binary avg, pass@1, pass@2, pass@4, pass@8, pass@16, pass@32, pass@64, pass@12[8]. Header: "tasks, n=128 rollouts per task/model, 12,288 rows per model."
Row "[b]ase Qwen3.5-35B-[A]3B": 96, 128, 0.642, 0.642, 0.823, 0.933, 0.980, 0.996, 1.000, 1.000, [1.00]
Row "[?]kpt200": 96, 128, 0.697, 0.697, 0.842, 0.930, 0.973, 0.991, 0.998, 1.000, [1.00]
Row "[?] extrap": 96, 128, 0.710, 0.710, 0.846, 0.934, 0.977, 0.992, 0.998, 1.000, [1.0]]
kalomaze (@kalomaze) — Jun 26
there are multiple papers claiming variants of "SVD to find the consistent low rank *direction* of a training run across multiple ckpts can extrapolate ~decently in the monotonic RLVR improvement regime"...
Note from Claude Sonnet 5
A benchmark results table for a Qwen3.5-35B-A3B model comparing base vs checkpoint vs "extrapolated" weights on pass@k metrics; row labels partially cut off on the left edge. Technical ML research tweet thread.
machine learning researchloramodel extrapolationrlvrtwitterqwen
León @LeonGuertler · 1h
3/7 Code Example
Using unstable baselines is relatively simple. Just import unstable (has to be installed via "pip install unstable-rl"), initialize the necessary objects and start training. To make training more accessible, beyond the computational (and vRAM) requirement reductions achieved by using LoRA, we implemented optional full activation check-pointing and training sequence truncation (i.e. only training on the first "N" tokens, but generating (and evaluating) sequences of up to "M" tokens; this works very well in practice). When using both of these, it is possible to train up to 4B parameter models using self-play on just 24gb of vRAM (although it is worth mentioning that currently you will need at least 2 GPUs (i.e. minimum of 2x24gb) since learners and actors (currently) can't share the same gpu).
This is also a good time to explain why we built this library "LoRA first". UnstableBaselines is supposed to be a hackable/fast prototyping library making it easy to not just iterate on ideas, but also research topics that would be harder to research w/o lora (i.e. opponent sampling). Furthermore, as shown by RAGEN/Tina/etc. reasoning (at least seemingly) can sufficiently be trained via low-rank updates.
[Partially visible code snippet below:]
import ray, unstable
import unstable.reward_transformations as retra
tracker = unstable.Tracker.options(name="Tracker").remote(run_name="demo", wandb_project="UB")
step_buffer = unstable.StepBuffer.options(name="StepBuffer").remote(...
Note from Claude Sonnet 5
A Twitter thread (3 of 7) from León Guertler introducing "UnstableBaselines"/"unstable-rl", a hackable LoRA-first RL self-play training library, discussing VRAM requirements and low-rank (LoRA) reasoning training. Technical ML tooling content relevant to Nathan's RL/training interests.
reinforcement learningloraself-playml toolingtwitteropen source