data efficiency

Samip @industriaalist · Apr 19 quick writeup on why i think diffusion isn't more data efficient than AR, since it seemed to surprise a lot of people: - the case for diffusion > AR ([1], [2]) rests on AR saturating at <5 epochs while diffusion can be trained for hundreds of epochs without overfitting. but that's AR with default regularization. with Slowrun we train AR for >30 epochs without overfitting using heavy regularization (15x standard weight decay and dropout), which captures the gains diffusion gets over hundreds of epochs. you can't push reg this hard on diffusion, the objective is already effectively regularizing the network - data augmentation is another lever that helps AR models: sequence permutation and token masking close a lot of the gap even without heavy regularization - [3] verifies this cleanly: simple dropout, weight decay, and token masking were enough to bridge the gap and even *surpass* diffusion. aligns with what we've seen [1] arxiv.org/abs/2511.03276 [2] arxiv.org/abs/2507.15857 [3] arxiv.org/abs/2510.04071 [Link card] arxiv.org — Diffusion Language Models are Super Data Learners

Harry Partridge @part_harry_ Pretraining is data-inefficient. This is entirely a consequence of the fact that we throw away the KV cache after every forward-backward step! If we can integrate efficient KV cache compaction into pretraining, we will unlock human level data efficiency. Neural KV cache compaction makes this possible. [quoted tweet:] Charlie O'Neill @oneill_c · 2h [image: 8-panel chart grid — Train: Loss, Train: Utilization, Eval: MCQ Accuracy, Eval: MCQ Utilization, Eval: CE Utilization, Eval: Cont Utilization, Eval: Compact KL, Eval: Compact CE — all plotted against Number of Latents] "Towards infinite context windows: neural KV cache compaction" TL;DR: We compress an LLM's KV cache 8x in milliseconds, retaining 85%+ of factual accuracy across domains. Here's how. Introduction... 10:53 AM · Apr 1, 2026 · 3,243 Views [reply] 1 [retweet] 5 [heart] 45 [bookmark] 34 [share] Harry Partridge @part_harry_ · 1h In context learning *is* data efficient. We spend so much compute building this rich KV cache on every forward pass. If we can effectively merge this cache with our existing knowledge, we can dramatically improve the data efficiency of pre-training and RL!

Percy Liang @percyliang In our last episode, careful tuning, scaling, and ensembles led to a 5x gain in data efficiency (requires 5x less data to get the same loss). Now, with a rephraser model, we can get an additional 1.8x gain in data efficiency. I know, everyone's compute constrained, but we're preparing for a data-constrained future. > QUOTED: Konwoo Kim @konwookim · 7h > for data-constrained pre-training, synth data isn't just benchmaxxing, it lowers loss on the real data distribution as we generate more tokens > for even better scaling, treat synth gens as forming on... > [Embedded paper image: "Data-efficient pre-training by scaling synthetic megadocs" — Konwoo Kim, Suhas Kotha, Yejin Choi, Tatsunori Hashimoto, Nick Haber, Percy Liang, Stanford University. Diagrams of "Simple Rephrasing," "Stitched Rephrasing," and "Latent Thoughts" methods for generating synthetic training data from real docs; a scaling chart "Scaling Generations" showing IID Loss decreasing from 3.55 (real data only) down to ~3.34-3.41 as generations per pretraining doc increase from 0 to 32, with Latent Thoughts performing best.] 2:52 PM · Mar 20, 2026 · 5,748 Views