[top of screen, author header cut off]
Following our prior work on reasoning with sampling, we now introduce sampling to the posttraining stack.
1/n
[attached GIF/figure, shown as a slide]
Finetuning with Sampling:
SFT Learns Better Than You Think
Aayush Karan¹, Sitan Chen¹, Yilun Du¹
¹Harvard University
[left panel: bar chart, y-axis Accuracy (%), 20 to 70; groups Chemistry, MMLU, AMC]
Chemistry: 61.8, 61.8, 66.0
MMLU: 58.6, 65.1, 69.2
AMC: 27.7, 34.9, 37.4
[right panel: scatter, y-axis Prior Task Retention (%), 85 to 100; x-axis New Task Accuracy (%), 30 to 70; one point labelled Base near the top left]
[legend] SFT OPSD (On-Policy) Ours (Sampling SFT)
60 replies · 264 reposts · 1.8K likes · 258K views
Yifei Wang @yifeiwang77 · 17h
Interesting wok and happy to see MCMC back to life. Have u tried DFT btw? It's also a way to make SFT more "on-policy" and can often get 5-10% gains on these stem tasks. One can also think DFT as a special token-level resampling method with weight p(x). And it would be cheaper w/o sampling.
[5 likes · 253 views]
Aayush Karan @aakaran31 · 2h
Yes! The key difference is that DFT reweights the SFT objective to fit the off-policy data, while we're shaping the data itself so that SFT is on-policy. I think actually having the concrete on-policy data is crucial for getting this to work well, since DFT can sacrifice generalization in service of less forgetting as the original OPSD/SDFT paper reports.
Note from Claude Sonnet 5
X thread announcing a Harvard paper, 'Finetuning with Sampling: SFT Learns Better Than You Think' (Karan, Chen, Du), with a bar chart comparing SFT, OPSD and sampling-SFT on Chemistry, MMLU and AMC, plus a reply exchange with Yifei Wang about DFT versus making SFT on-policy.
sftsamplingpost-trainingon-policymcmcmachine learning
Mathelirium ✓ @mathelirium · 5h
Now that we've seen a single random-walk MCMC get trapped in one valley and pretend that's the whole posterior, this scene shows one of the nicest fixes called Parallel Tempering.
Instead of one lonely chain at the true temperature, we run a ladder of copies at different "heat levels" on the same landscape. The cold chain still sees the sharp, deep wells, but the hotter chains see a flattened version where barriers are lower and it's easy to wander between basins.
Every few steps we let neighbouring temperatures swap states, so when a hot chain discovers a distant well it can hand that discovery back down to the cold chain. Visually you see faint hot blobs roaming the whole surface while the dark cold chain suddenly starts teleporting between wells it could never reach on its own. This is a geometry-aware algorithm.
#MCMC #ParallelTempering #BayesianInference #ComputationalStatistics #MonteCarlo #MultimodalPosteriors #HighDimensionalSpace
[Embedded video, paused at 0:51: animated 3D visualization titled "PARALLEL TEMPERING - HOT CHAINS, COLD POSTERIOR / Cold chain (β = 1) visiting multiple wells via swaps with hotter chains" — shows a 3D landscape with a wandering trajectory over multiple wells, a density histogram plot of target vs empirical distribution from the cold chain, a 2D scatter plot of posterior samples with cold chain path, and a step plot of which mode the cold chain occupies over time.]
Note from Claude Sonnet 5
An educational thread/animation explaining the Parallel Tempering MCMC algorithm for sampling multimodal posterior distributions. General computational statistics content, not directly tied to AI safety/alignment threads, though MCMC and posterior sampling techniques are occasionally relevant background for Bayesian ML methods.
mcmcbayesian inferencestatisticsmachine learningmonte carlotwittermath
