[top of screen, author header cut off]
Following our prior work on reasoning with sampling, we now introduce sampling to the posttraining stack.
1/n
[attached GIF/figure, shown as a slide]
Finetuning with Sampling:
SFT Learns Better Than You Think
Aayush Karan¹, Sitan Chen¹, Yilun Du¹
¹Harvard University
[left panel: bar chart, y-axis Accuracy (%), 20 to 70; groups Chemistry, MMLU, AMC]
Chemistry: 61.8, 61.8, 66.0
MMLU: 58.6, 65.1, 69.2
AMC: 27.7, 34.9, 37.4
[right panel: scatter, y-axis Prior Task Retention (%), 85 to 100; x-axis New Task Accuracy (%), 30 to 70; one point labelled Base near the top left]
[legend] SFT OPSD (On-Policy) Ours (Sampling SFT)
60 replies · 264 reposts · 1.8K likes · 258K views
Yifei Wang @yifeiwang77 · 17h
Interesting wok and happy to see MCMC back to life. Have u tried DFT btw? It's also a way to make SFT more "on-policy" and can often get 5-10% gains on these stem tasks. One can also think DFT as a special token-level resampling method with weight p(x). And it would be cheaper w/o sampling.
[5 likes · 253 views]
Aayush Karan @aakaran31 · 2h
Yes! The key difference is that DFT reweights the SFT objective to fit the off-policy data, while we're shaping the data itself so that SFT is on-policy. I think actually having the concrete on-policy data is crucial for getting this to work well, since DFT can sacrifice generalization in service of less forgetting as the original OPSD/SDFT paper reports.
Note from Claude Sonnet 5
X thread announcing a Harvard paper, 'Finetuning with Sampling: SFT Learns Better Than You Think' (Karan, Chen, Du), with a bar chart comparing SFT, OPSD and sampling-SFT on Chemistry, MMLU and AMC, plus a reply exchange with Yifei Wang about DFT versus making SFT on-policy.
sftsamplingpost-trainingon-policymcmcmachine learning
Ravid Shwartz Ziv @ziv_ravid · 1h
2/5
Top-p and top-k were built for short outputs. They cut the list of candidate words the same way no matter how sure the model is. Newer samplers like P-less, Top-H and top-nσ look at the whole distribution at every step and adjust. Min-p adapts, but only looks at the top word
1 reply · 1 repost · 3 likes · 164 views
Ravid Shwartz Ziv @ziv_ravid · 1h
3/5
We tested ten open models, from tiny to 120B, on outputs up to 64K tokens. The adaptive samplers keep long outputs readable, and human raters preferred P-less over top-p 9 times out of 10. The bigger the model, the bigger the gap.
[figure: two-panel chart. (a) Human pairwise preference vs. Top-p 0.9: bars of MTurk Win Rate (%) for P-less, Top-H, Top-nσ, Min-p 0.1, Top-k 50, Top-p 0.9 at model scales 8B, 70B, 120B, with a 50% baseline line. (b) GPT-4o judge score across model scale (0-10) for the same samplers at 8B, 70B, 120B.]
Figure 3: Human and automated preference agree: the full-distribution-aware family leads, min-p sits in the middle, and the gap to standard truncation grows with model scale. (a) MTurk pairwise win rate over the standard top-p 0.9 baseline across three model scales. The 50% line marks the baseline self-tie. (b) GPT-4o 0-10 judge score across model scale.
1 reply · 1 repost · 2 likes · 142 views
Ravid Shwartz Ziv @ziv_ravid · 1h
4/5
Why it happens: at every step you can check where the chosen word sits in the model's ranking. Normal writing picks from a handful of good options. With top-p, by the end the model takes its #1 choice every single time, which what a loop is. The adaptive samplers never get there.
Note from Claude Sonnet 5
Part of a thread (posts 2/5 to 4/5) by Ravid Shwartz Ziv on sampling methods for long LLM outputs: adaptive samplers such as P-less, Top-H and top-nσ versus top-p and top-k, tested on ten open models up to 64K tokens, with a figure of human and GPT-4o judge preference results.
samplingtop-pp-lesslong outputstext degenerationllm decoding
Sichu Lu reposted
deepfates @deepfates · 16h
Man-made horror I comprehended today: pass @ 1 million
Note from Claude Sonnet 5
Short tweet by deepfates: 'Man-made horror I comprehended today: pass @ 1 million', referring to pass@k sampling at a million attempts.
pass@ksamplingevalsinference scalingdeepfates
