— web clipping, 4,409 words — published 2026-04-01
Towards infinite context windows: neural KV cache compaction
TL;DR: We compress an LLM's KV cache 8x in milliseconds, retaining 85%+ of factual accuracy across domains. Here's how. # Introduction Current LLMs have a memory problem. Not the kind solved by longer context windows, but the kind where an agent forgets everything at the end of a conversation. Today's approaches to persistent memory are crude. They are either lossless […]
Summary by Claude Opus 5
Charlie O'Neill on STILL, a perceiver bottleneck that compresses a frozen LLM's KV cache 8x in milliseconds while keeping 85%+ factual accuracy, trained per layer with learned query vectors cross-attending into the full cache. Framed as the first step toward an intermediate memory layer for continual learning, and motivated explicitly by analogy to what SAEs did for dictionary learning.
Full text not reproduced here — kept as Towards infinite context windows neural KV cache compaction.md in Nathan's clippings archive.