5 captures, most recent first.
fellow ⚗ traveler 🔥 (@architectonyx) — 5h
in the 80s, there was this nice idea of the brain being "holographic", with information stored nonlocally
this was motivated by appeals to Fourier analysis, but interestingly, i think function approximation alone gets you there
[Embedded video/image, 0:11 duration, showing a blurred/noisy gradient panel on the left and a grid of small sample images (landscapes, animals, fruit, objects) on the right]
Note from Claude Sonnet 5
Tweet with an embedded short video (paused at 0:11) illustrating a holographic/distributed-representation visualization alongside a mosaic of sample training images.
neurosciencemachine learningrepresentation learningtwitter
catid @MrCatid · 9h
So now that we know intelligence (planning, reasoning, etc) is just finding efficient representations, who's working on a new type of ML model that learns representations? Seems like we need something new closer to VQVAE with an evolutionary rule
Note from Claude Sonnet 5
A tweet speculating about future ML architectures for representation learning (VQVAE-like with evolutionary rules). Technical AI research commentary, tangentially related to Nathan's own architecture work on brain_graph_1.
twittermachine learningrepresentation learningvqvaearchitecture research
diff between future and predicted future states is reward
[Embedded diagram: same ICM figure as previous screenshot — forward model predicting φ̂(s_t+1), compared (−) against actual φ(s_t+1) to produce intrinsic reward r^i_t; inverse model predicts action â_t from φ(s_t), φ(s_t+1).]
💬1 🔁 ❤️4 📊218 🔖 ⤴
Joey (e/λ) @shxf0072 · 2h
forward model is trying to predict future latent given current so it works like curiosity
while inverse model teachs what to ignore
encoder learns to ignore noise like trees leave since it doesn't help inverse models predict which action was taken
[Embedded diagram, repeated: full ICM architecture — agent state s_t, policy π producing action a_t, environment E transitioning to s_t+1, action a_t+1 via π, ICM module consuming s_t/a_t/s_t+1 to produce intrinsic reward r^i_t, combined with extrinsic reward r^e_t.]
Note from Claude Sonnet 5
Continuation of the same ICM (Intrinsic Curiosity Module) thread — explains the intuition that the inverse dynamics model (predicting the action taken from consecutive states) forces the feature encoder to discard action-irrelevant noise (like windblown leaves), so the forward model's prediction error in that cleaned feature space becomes a principled curiosity/novelty signal. Same relevance as the previous screenshot: architecture pattern applicable to brain_graph_1's intrinsic-motivation/curiosity design.
reinforcement learningcuriosity-driven explorationintrinsic rewardicmrepresentation learningbrain_graph_1
James Chen @jchencxh · Mar 19
The actual implementation I give for doing this on a MIM task is actually very straightforward.
You just take the hidden representations after every level of a ViT, sample some subset, and use each representation separately to predict masked patches at every single level.
1 reply, 9 likes, 454 views
James Chen @jchencxh · 23h
There are actually two reasons for predicting lower level abstractions:
1) less spuriousness (harder to be spurious for predicting a hierarchy)
2) retain lower level reps that can be useful for later composition in the net
Note that for 2), it's because we're only communicating across representations levels through a single embedding through a feed forward net.
If there was a larger communication bandwidth across the representation levels (e.g., some version of attention residuals), this problem could go away, and there's a lot more possibility for the higher level abstractions we can compose (as it's becomes easier to compose low and very high levels of abstraction, where the very lower level of abstraction would not be kept in a standard feed forward structure).
I think that a newer class of architectures is probably better to exploit this type of learning over the hierarchy.
Note from Claude Sonnet 5
A technical ML architecture thread on masked-image-modeling (MIM) with per-layer prediction of masked patches from a Vision Transformer (ViT), discussing hierarchical representation learning and cross-level communication bandwidth. Likely research reading relevant to Nathan's own architecture work on brain_graph_1 / latent iteration design.
twittermachine learning researchvision transformermasked image modelingrepresentation learningneural architecture
Kenneth Sta... @kenneth0st... · 14h
Carl Jung made a point long ago that both foreshadows fractured entangled representation (FER) and offers a thought-provoking critique of modern ML in general: "Beware of unearned wisdom." (I'd update it to "unearned knowledge" for AI today.)
If the way that you acquire knowledge impacts your facility for applying that knowledge in the future through its consequent underlying representation, then what price do you pay for the unnatural vacuuming up of vast swaths of knowledge in a giant disorganized batch?
Unearned knowledge has a cost that's rarely if ever discussed in AI or ML.
Thank you to @jakobmrees, an undergrad at NYU, for perceptively bringing this quote to my attention!
Note from Claude Sonnet 5
A tweet arguing that LLM pretraining's mode of "unearned" knowledge acquisition (bulk, disorganized ingestion vs. earned/structured learning) may degrade the quality/organization of internal representations, drawing on a Jung quote. Conceptually adjacent to Nathan's interest in how training methodology shapes model self-models/representations (cf. his "compelled vs endogenous values" and RLHF-representation notes), though from an ML-architecture rather than welfare angle.
twittermachine learningrepresentation learningjungpretrainingepistemics