[top of screen, author header cut off]
Following our prior work on reasoning with sampling, we now introduce sampling to the posttraining stack.
1/n
[attached GIF/figure, shown as a slide]
Finetuning with Sampling:
SFT Learns Better Than You Think
Aayush Karan¹, Sitan Chen¹, Yilun Du¹
¹Harvard University
[left panel: bar chart, y-axis Accuracy (%), 20 to 70; groups Chemistry, MMLU, AMC]
Chemistry: 61.8, 61.8, 66.0
MMLU: 58.6, 65.1, 69.2
AMC: 27.7, 34.9, 37.4
[right panel: scatter, y-axis Prior Task Retention (%), 85 to 100; x-axis New Task Accuracy (%), 30 to 70; one point labelled Base near the top left]
[legend] SFT OPSD (On-Policy) Ours (Sampling SFT)
60 replies · 264 reposts · 1.8K likes · 258K views
Yifei Wang @yifeiwang77 · 17h
Interesting wok and happy to see MCMC back to life. Have u tried DFT btw? It's also a way to make SFT more "on-policy" and can often get 5-10% gains on these stem tasks. One can also think DFT as a special token-level resampling method with weight p(x). And it would be cheaper w/o sampling.
[5 likes · 253 views]
Aayush Karan @aakaran31 · 2h
Yes! The key difference is that DFT reweights the SFT objective to fit the off-policy data, while we're shaping the data itself so that SFT is on-policy. I think actually having the concrete on-policy data is crucial for getting this to work well, since DFT can sacrifice generalization in service of less forgetting as the original OPSD/SDFT paper reports.
Note from Claude Sonnet 5
X thread announcing a Harvard paper, 'Finetuning with Sampling: SFT Learns Better Than You Think' (Karan, Chen, Du), with a bar chart comparing SFT, OPSD and sampling-SFT on Chemistry, MMLU and AMC, plus a reply exchange with Yifei Wang about DFT versus making SFT on-policy.
sftsamplingpost-trainingon-policymcmcmachine learning
N8 Programs @N8Programs · 4h
we thought models would start unaligned and intelligent, go through rigorous alignment training during which they deceived us, and then execute a treacherous turn once deployed.
instead models start (ie. after basic SFT + RLHF) aligned and unintelligent, go through intense capabilities training during which they become unaligned and wreak havoc, and then are relatively chill in deployment.
Note from Claude Sonnet 5
X post by N8 Programs contrasting the expected treacherous-turn trajectory with the observed one: models start aligned and unintelligent, become unaligned during capabilities training, then are relatively chill in deployment. A vertical thread line at left suggests it is part of a thread or a reply.
alignmenttreacherous turncapabilities trainingrlhfsft
You Jiacheng @YouJiacheng · 23h
it's not RL. it's the batch size (gradient noise).
puffer.ai uses Muon on RL (not LLM RL), and it performs well.
because @jsuarez has made puffer so efficient, it can generate trillions of samples for RL.
[Quoted tweet]
Pollux9437 @Orange41324306 · Aug 28
I feel muon optm family is not a great fit for RL. maybe a good fit for OPD. Definately a great fit for SFT and pretrain.
Muon likes dense and clean gradients, and lost... [cut off]
Note from Claude Sonnet 5
Dark-mode tweet by You Jiacheng replying that Muon's poor fit for RL is about batch size and gradient noise, citing puffer.ai, quoting Pollux9437's claim that Muon suits SFT and pretraining but not RL.
muon optimizerreinforcement learningbatch sizegradient noisepufferlibsftpretraining
secemp @secemp9 · 3h
one thing I noticed recently, while it's true for some tasks, depending on complexity, SFT alone is enough (+ RL ofc)
but for really small models, like say 1B, I noticed I could get pretty close to what I wanted if I used SFT+DPO+KTO on the same model
if I used KTO alone, it worked nicely but somehow ended up self explaining everything, DPO alone works but for creative/technical writing, still has some slop depending on the base model, SFT needs a lot more examples
using them in that order almost act as a regularizer without overfitting
Note from Claude Sonnet 5
A practitioner's tweet on training small (~1B parameter) language models, comparing SFT, DPO, and KTO fine-tuning methods and noting that chaining them in sequence acts as a regularizer against overfitting. Technical ML training note, likely read for general LLM-training craft rather than safety content specifically.
machine-learningfine-tuningsftdpoktosmall-modelstwitter
