percy liang

Percy Liang @percyliang In our last episode, careful tuning, scaling, and ensembles led to a 5x gain in data efficiency (requires 5x less data to get the same loss). Now, with a rephraser model, we can get an additional 1.8x gain in data efficiency. I know, everyone's compute constrained, but we're preparing for a data-constrained future. > QUOTED: Konwoo Kim @konwookim · 7h > for data-constrained pre-training, synth data isn't just benchmaxxing, it lowers loss on the real data distribution as we generate more tokens > for even better scaling, treat synth gens as forming on... > [Embedded paper image: "Data-efficient pre-training by scaling synthetic megadocs" — Konwoo Kim, Suhas Kotha, Yejin Choi, Tatsunori Hashimoto, Nick Haber, Percy Liang, Stanford University. Diagrams of "Simple Rephrasing," "Stitched Rephrasing," and "Latent Thoughts" methods for generating synthetic training data from real docs; a scaling chart "Scaling Generations" showing IID Loss decreasing from 3.55 (real data only) down to ~3.34-3.41 as generations per pretraining doc increase from 0 to 32, with Latent Thoughts performing best.] 2:52 PM · Mar 20, 2026 · 5,748 Views