Artur Chakhvadze @norpadon
— quoting @jxmnop (jxmo)
Artur Chakhvadze @norpadon · 22h
A trick I came up with in ~2017 in the context of image matting is that you can train a network to predict d_loss/d_y instead of y and do a gradient descent for multiple iterations during inference. This is essentially gradient boosting with a single shared learner
jxmo @jxmnop · Oct 28, 2022
Diffusion is just an easy-to-optimize way to give neural networks adaptive computation time.
Makes sense then that diffusion mode... [cut off]
Note from Claude Sonnet 5
A tweet on an architectural trick — training a network to predict the gradient of the loss (d_loss/d_y) and applying iterative gradient descent at inference, framed as "gradient boosting with a single shared learner" — quoting an older tweet framing diffusion models as adaptive computation time. Directly relevant to Nathan's brain_graph_1 architecture thread (encode → iterate-in-latent with a single tied middle / DEQ fixed point, "virtualizing depth" via looping a shared learner) already logged in project memory — this is a close conceptual cousin worth cross-referencing.
twittermachine-learningarchitecturediffusion-modelsadaptive-computationgradient-boostingdeqinterpretability