Sauers @Sauers_
Sauers @Sauers_
Our blessed manifolds vs their barbarous shattered features
[Image: meme diptych — left, a muscular "strong doge" with a smooth continuous color-wheel sphere on its torso; right, a scrawny "weak doge" surrounded by scattered discrete colored spheres. Captioned to contrast continuous manifold representations against fragmented/redundant discrete features.]
Ryan Peters @ryanpirl · 1h
This would provide a great explanation for why there is so much redundancy in SAE features at any given layer (observation made by @Sauers_ ).
For example, if you search through the Qwen3-4b ...
[Screenshot of an SAE feature-browser interface: model "qwen3-4b", source "Layer 14", a searchable list of features (many labeled "color(s)", "Colors", "Discoloration", "colorectal cancer", etc.), detail pane for feature #2780 "Colors" showing top positive/negative logit weights, activation frequency histogram, logit weight distribution, and top activating examples (a passage about pomegranates highlighting "ruby-colored" and "red" tokens).]
5:35 PM · May 21, 2026 · 80 Views
Note from Claude Sonnet 5
Interpretability-research tweet arguing that sparse autoencoder (SAE) feature redundancy arises because true representations live on continuous manifolds that SAEs shatter into many overlapping discrete features (illustrated via a color-wheel meme), with a concrete example browsing Qwen3-4b's "Colors" feature. Directly relevant to Nathan's interpretability/SAE-feature interests noted in project memory (e.g. GoodFire deception/self-awareness features).
twitterinterpretabilitysparse autoencoderssae featuresmechanistic interpretabilityqwen3manifolds