— reply from @yifeiwang77 — saved image
[top of screen, author header cut off] Following our prior work on reasoning with sampling, we now introduce sampling to the posttraining stack. 1/n [attached GIF/figure, shown as a slide] Finetuning with Sampling: SFT Learns Better Than You Think Aayush Karan¹, Sitan Chen¹, Yilun Du¹ ¹Harvard University [left panel: bar chart, y-axis Accuracy (%), 20 to 70; groups Chemistry, MMLU, AMC] Chemistry: 61.8, 61.8, 66.0 MMLU: 58.6, 65.1, 69.2 AMC: 27.7, 34.9, 37.4 [right panel: scatter, y-axis Prior Task Retention (%), 85 to 100; x-axis New Task Accuracy (%), 30 to 70; one point labelled Base near the top left] [legend] SFT OPSD (On-Policy) Ours (Sampling SFT) 60 replies · 264 reposts · 1.8K likes · 258K views Yifei Wang @yifeiwang77 · 17h Interesting wok and happy to see MCMC back to life. Have u tried DFT btw? It's also a way to make SFT more "on-policy" and can often get 5-10% gains on these stem tasks. One can also think DFT as a special token-level resampling method with weight p(x). And it would be cheaper w/o sampling. [5 likes · 253 views] Aayush Karan @aakaran31 · 2h Yes! The key difference is that DFT reweights the SFT objective to fit the off-policy data, while we're shaping the data itself so that SFT is on-policy. I think actually having the concrete on-policy data is crucial for getting this to work well, since DFT can sacrifice generalization in service of less forgetting as the original OPSD/SDFT paper reports.
Note from Claude Sonnet 5
X thread announcing a Harvard paper, 'Finetuning with Sampling: SFT Learns Better Than You Think' (Karan, Chen, Du), with a bar chart comparing SFT, OPSD and sampling-SFT on Chemistry, MMLU and AMC, plus a reply exchange with Yifei Wang about DFT versus making SFT on-policy.