kalomaze @kalomaze
— saved image
kalomaze @kalomaze · 8h so i have experiments that show pretty convincingly, tensor-train style decomposition works for modeling exact AR joints over high dimensional data tractably. i did lit search and only found recent papers which apply it very narrowly, without realizing the larger implications 9 replies, 4 reposts, 148 likes, 6.7K views kalomaze @kalomaze · 8h what i am trying to say is "next token prediction" can be generalized to "next joint prediction", generically, WITHOUT diffusion or MSE regression or flow matching a path exists for exact likelihood + policy gradients over far larger action spaces than you'd expect 1 reply, 1 repost, 38 likes, 1.1K views kalomaze @kalomaze · 8h arxiv.org/abs/1709.01662 more specifically, if you condition this kind of parameterization on a sufficiently rich transformer hidden state, you can optimize for exactly valid joints over combinatorially massive spaces, up to a rank bottleneck [Link card: arxiv.org — Unsupervised Generative Modeling Using Matrix Product States]
Note from Claude Sonnet 5
Twitter thread by kalomaze describing experiments showing tensor-train (matrix product state) decomposition can generalize 'next token prediction' to 'next joint prediction' over high-dimensional/combinatorial action spaces, enabling exact likelihood plus policy gradients without diffusion, MSE regression, or flow matching, citing arxiv paper 'Unsupervised Generative Modeling Using Matrix Product States' (1709.01662).
machine learningtwittertensor decompositiongenerative modelingreinforcement learningkalomaze