icm — Timeline

2026-05-03

diff between future and predicted future states is reward [Embedded diagram: same ICM figure as previous screenshot — forward model predicting φ̂(s_t+1), compared (−) against actual φ(s_t+1) to produce intrinsic reward r^i_t; inverse model predicts action â_t from φ(s_t), φ(s_t+1).] 💬1 🔁 ❤️4 📊218 🔖 ⤴ Joey (e/λ) @shxf0072 · 2h forward model is trying to predict future latent given current so it works like curiosity while inverse model teachs what to ignore encoder learns to ignore noise like trees leave since it doesn't help inverse models predict which action was taken [Embedded diagram, repeated: full ICM architecture — agent state s_t, policy π producing action a_t, environment E transitioning to s_t+1, action a_t+1 via π, ICM module consuming s_t/a_t/s_t+1 to produce intrinsic reward r^i_t, combined with extrinsic reward r^e_t.]

Note from Claude Sonnet 5

Continuation of the same ICM (Intrinsic Curiosity Module) thread — explains the intuition that the inverse dynamics model (predicting the action taken from consecutive states) forces the feature encoder to discard action-irrelevant noise (like windblown leaves), so the forward model's prediction error in that cleaned feature space becomes a principled curiosity/novelty signal. Same relevance as the previous screenshot: architecture pattern applicable to brain_graph_1's intrinsic-motivation/curiosity design.

reinforcement learning curiosity-driven exploration intrinsic reward icm representation learning brain_graph_1

Joey (e/λ @shxf0072

2026-05-03

Joey (e/λ) @shxf0072 this example is from rl paper, one of the best one curiosity driven exploration, if you squint eyes hard enough you can find jepa like arch trained end to end with rl [Embedded image: page from a paper, "Curiosity-driven Exploration by Self-supervised Prediction" — Figure 2 diagram of an agent interacting with environment via policy π, ICM (Intrinsic Curiosity Module) computing intrinsic reward r^i from states s_t, s_t+1; right side shows ICM internals: forward model predicting φ̂(s_t+1) from φ(s_t) and a_t, inverse model predicting â_t from φ(s_t), φ(s_t+1), compared against actual a_t. Caption explains: agent optimizes sum of extrinsic reward r^e and curiosity-based intrinsic reward r^i; prediction error in feature space (not pixel space) is the curiosity signal, making exploration robust to uncontrollable environment aspects. Below, section 2.1 "Prediction error as curiosity reward" discusses why raw pixel-space prediction is undesirable (e.g. unpredictable leaf motion causing an "artificial curiosity trap"), and the need for a feature space that models controllable/agent-affecting elements while ignoring irrelevant background variation. Two passages hand-underlined in red: the paragraph about prediction error being unclear as an objective, and the passage about no known computationally feasible mechanism for measuring learning progress instead of prediction error.]

Note from Claude Sonnet 5

A tweet highlighting the classic Intrinsic Curiosity Module (ICM) paper (Pathak et al., "Curiosity-driven Exploration by Self-supervised Prediction") as an early example of a JEPA-like (joint embedding predictive architecture) approach trained end-to-end with RL — predicting in learned feature space rather than raw pixels to generate intrinsic/curiosity reward. Directly relevant to Nathan's brain_graph_1 project, which uses PVLV dopamine and curiosity-style intrinsic motivation signals; ICM's forward/inverse-model feature-space prediction is a foundational architecture pattern for that kind of RL exploration design.

reinforcement learning curiosity-driven exploration intrinsic reward jepa self-supervised learning icm brain_graph_1 ai research paper