— saved image
Ravid Shwartz Ziv @ziv_ravid · 1h 2/5 Top-p and top-k were built for short outputs. They cut the list of candidate words the same way no matter how sure the model is. Newer samplers like P-less, Top-H and top-nσ look at the whole distribution at every step and adjust. Min-p adapts, but only looks at the top word 1 reply · 1 repost · 3 likes · 164 views Ravid Shwartz Ziv @ziv_ravid · 1h 3/5 We tested ten open models, from tiny to 120B, on outputs up to 64K tokens. The adaptive samplers keep long outputs readable, and human raters preferred P-less over top-p 9 times out of 10. The bigger the model, the bigger the gap. [figure: two-panel chart. (a) Human pairwise preference vs. Top-p 0.9: bars of MTurk Win Rate (%) for P-less, Top-H, Top-nσ, Min-p 0.1, Top-k 50, Top-p 0.9 at model scales 8B, 70B, 120B, with a 50% baseline line. (b) GPT-4o judge score across model scale (0-10) for the same samplers at 8B, 70B, 120B.] Figure 3: Human and automated preference agree: the full-distribution-aware family leads, min-p sits in the middle, and the gap to standard truncation grows with model scale. (a) MTurk pairwise win rate over the standard top-p 0.9 baseline across three model scales. The 50% line marks the baseline self-tie. (b) GPT-4o 0-10 judge score across model scale. 1 reply · 1 repost · 2 likes · 142 views Ravid Shwartz Ziv @ziv_ravid · 1h 4/5 Why it happens: at every step you can check where the chosen word sits in the model's ranking. Normal writing picks from a handful of good options. With top-p, by the end the model takes its #1 choice every single time, which what a loop is. The adaptive samplers never get there.
Note from Claude Sonnet 5
Part of a thread (posts 2/5 to 4/5) by Ravid Shwartz Ziv on sampling methods for long LLM outputs: adaptive samplers such as P-less, Top-H and top-nσ versus top-p and top-k, tested on ten open models up to 64K tokens, with a figure of human and GPT-4o judge preference results.
samplingtop-pp-lesslong outputstext degenerationllm decoding