diffusion models

Siddharth Ancha @siddancha · 19h This is very cool! But also not that surprising. Flow matching models are "uniquely identifiable" i.e. any sufficiently well trained flow model on the same data distribution will learn the same latent encoding (mapping from z → x), regardless of architecture. In @jaschasd's words: youtube.com/watch?v=XCUlnH... . This should be true for diffusion models too if you properly seed the randomness used to generate intermediate samples. It's still remarkable how strongly identifiable flow matching models are, especially the male/female and CelebHQ/FFHQ experiments. Says a lot about the datasets! > QUOTED: > Kwang Moo Yi @kwangmoo_yi · Apr 20 > Briq et al., "The Amazing Stability of Flow Matching" > > The attached image explains it all (with minor caption error though) -- training flow matching ... > > [Figure 1: Stability of the generated images grid, 4 panels: (a) Two disjoint random subsets — model trained on two disjoint random subsets of data produces visually very similar images; (b) DiT-XL/4 → DiT-S/2 → U-Net — model capacity/architecture change retains high similarity; (c) Both genders/Female/Male — data split by zero-shot gender classification, retained partition preserves semantic interpretation while complementary class swaps it; (d) CelebHQ → FFHQ — changing training dataset while keeping the same VAE retains similarity too. Caption: "Figure 1: Stability of the generated images. (a) We train the model on two disjoint random subsets of the data, and obtain visually very similar images. (b) The data is split into two sets based on zero-shot classification as male/female. Images we visually interpret as belonging to the retained partition are semantically preserved, while images of the complementary class swap the semantic interpretation. (c) Model capacity change from DiT-XL to DiT-S retains high similarity, while switching to a U-Net architecture retains similarity to a lesser degree. (d) Changing the training dataset from CelebHQ to FFHQ, while still using CelebHQ VAE, retains similarity too."]

Samip @industriaalist · Apr 19 quick writeup on why i think diffusion isn't more data efficient than AR, since it seemed to surprise a lot of people: - the case for diffusion > AR ([1], [2]) rests on AR saturating at <5 epochs while diffusion can be trained for hundreds of epochs without overfitting. but that's AR with default regularization. with Slowrun we train AR for >30 epochs without overfitting using heavy regularization (15x standard weight decay and dropout), which captures the gains diffusion gets over hundreds of epochs. you can't push reg this hard on diffusion, the objective is already effectively regularizing the network - data augmentation is another lever that helps AR models: sequence permutation and token masking close a lot of the gap even without heavy regularization - [3] verifies this cleanly: simple dropout, weight decay, and token masking were enough to bridge the gap and even *surpass* diffusion. aligns with what we've seen [1] arxiv.org/abs/2511.03276 [2] arxiv.org/abs/2507.15857 [3] arxiv.org/abs/2510.04071 [Link card] arxiv.org — Diffusion Language Models are Super Data Learners

In the very beginning of diffusion training, you can plot gradient std, variance, magnitude, MSE-loss, gradient noise-scale (critical batch size estimation) for different timestep and its beautiful actually [image: six-panel plot grid — Gradient Variance vs Diffusion Timestep, Gradient Std vs Diffusion Timestep, Gradient Noise Scale vs Diffusion Timestep (all showing bell-curve/rising shapes peaking mid-range), Mean Gradient Magnitude vs Diffusion Timestep (sigmoid rise), Average MSE Loss vs Diffusion Timestep (declining sigmoid), and Noise Scale vs Variance colored by timestep (scatter, arc shape)]