Samip @industriaalist
Samip @industriaalist · Apr 19
quick writeup on why i think diffusion isn't more data efficient than AR, since it seemed to surprise a lot of people:
- the case for diffusion > AR ([1], [2]) rests on AR saturating at <5 epochs while diffusion can be trained for hundreds of epochs without overfitting. but that's AR with default regularization. with Slowrun we train AR for >30 epochs without overfitting using heavy regularization (15x standard weight decay and dropout), which captures the gains diffusion gets over hundreds of epochs. you can't push reg this hard on diffusion, the objective is already effectively regularizing the network
- data augmentation is another lever that helps AR models: sequence permutation and token masking close a lot of the gap even without heavy regularization
- [3] verifies this cleanly: simple dropout, weight decay, and token masking were enough to bridge the gap and even *surpass* diffusion. aligns with what we've seen
[1] arxiv.org/abs/2511.03276
[2] arxiv.org/abs/2507.15857
[3] arxiv.org/abs/2510.04071
[Link card] arxiv.org — Diffusion Language Models are Super Data Learners
Note from Claude Sonnet 5
A technical ML thread arguing that diffusion language models' apparent data efficiency advantage over autoregressive (AR) models is mostly an artifact of under-regularized AR baselines — heavy weight decay/dropout, sequence permutation, and token masking close or reverse the gap. Relevant to general ML architecture research Nathan follows (adjacent to brain_graph_1/DEQ architecture interests, though not directly cited there).
machine learningdiffusion modelsautoregressive modelsdata efficiencytwittersamiparxiv