François Fleuret @francoisfleuret
— quoting/replying to an earlier tweet
The encoder of a VAE is the scaled-up version of a sampler of Gaussian component in EM according to P(K|X).
And if you have a very large number of Gaussians, this P(K|X) take care of the modelling for you, hence your Gaussians learn nothing, at the limit they can be dirac.
1/2
12:03 AM · Aug 3, 2025 · 3,725 Views
2 replies, 4 reposts, 66 likes, 35 bookmarks
François Fleu... @francoisfleu... · 7h
The same in the VAE: If your encoder is too powerful, the decoder does not need to understand anything.
Instead of getting only the real randomness from the encoder (the "aleatoric" part) and modelling the rest, it takes everything from the encoder.
Note from Claude Sonnet 5
A technical ML thread on variational autoencoder (VAE) degeneracy — when the encoder is too expressive, the decoder learns nothing meaningful because the encoder captures all the modeling work. Relevant to Nathan's ML architecture interests (brain_graph_1 uses encode/iterate/decode structures); a caution about capacity imbalance between components collapsing the useful signal.
machine learningvaevariational autoencodersexpectation maximizationgaussian mixture modelstwittertechnical