kv cache

Han Xiao (@hxiao) · 7h: "finally got qwen3.5-35b-a3b running at 1M context on a single L4 24GB, with 905K tokens prefilled (86% full). decoding at 9 tok/s. stuffed the entire 天龙八部 (a 1.2M-word jin yong novel, 50 chapters) into KV cache as a test. the goal is to use long-context LLM + KV cache on a budget GPU as a document store, to enbale vectorless search - no embeddings, no chunking, just LLM+raw context. L4 is the most popular budget GPU in prod, and qwen3.5-35b-a3b is the only model where 1M context can theoretically fit on 24GB VRAM. 27B dense won't work. extremely tight budget. what it took to make it fit: • YaRN RoPE scaling (262K training context → 1M) • Q3_K_M quantization (20GB → 15.2GB) • turbo3 KV cache (23GB → 4GB for 1M tokens) • native turbo3 flash attention via Madreag/turbo3-cuda (no fp16 decompression, saves ~1.7GB that would OOM) • ubatch=128 (compute buffer 3GB → 779MB) • slot save/restore (68-min cold prefill → 2.3s restore) • append_to_slot patch (append query directly to cached 905K KV state, skip resending the whole novel) • patches for qwen3.5's hybrid attention+recurrent architecture (KV truncation + slot restore were broken). this setup gives peak vram at 96% utilization (22,052 MiB / 23,034 MiB)" [Embedded card/screenshot: "Qwen3.5-35B-A3B 1M Context on L4 24GB — Run Qwen3.5-35B-A3B with 1 million token context on a single NVIDIA L4 (24GB VRAM) using TurboQuant extreme KV cache quantization. Minimum hardware: 1x NVIDIA L4 24GB VRAM, CUDA 12.8, 16GB RAM, 20GB disk. Key Results table: Full context length 1,048,576 tokens (1M); Prefilled context length 904,949 tokens (86.3%); Model quantization Q3_K_M (3.51 bpw); Thinking output Disabled (--reasoning off); Model weights 15,190 MiB; ...(table cut off)]

Harry Partridge @part_harry_ Pretraining is data-inefficient. This is entirely a consequence of the fact that we throw away the KV cache after every forward-backward step! If we can integrate efficient KV cache compaction into pretraining, we will unlock human level data efficiency. Neural KV cache compaction makes this possible. [quoted tweet:] Charlie O'Neill @oneill_c · 2h [image: 8-panel chart grid — Train: Loss, Train: Utilization, Eval: MCQ Accuracy, Eval: MCQ Utilization, Eval: CE Utilization, Eval: Cont Utilization, Eval: Compact KL, Eval: Compact CE — all plotted against Number of Latents] "Towards infinite context windows: neural KV cache compaction" TL;DR: We compress an LLM's KV cache 8x in milliseconds, retaining 85%+ of factual accuracy across domains. Here's how. Introduction... 10:53 AM · Apr 1, 2026 · 3,243 Views [reply] 1 [retweet] 5 [heart] 45 [bookmark] 34 [share] Harry Partridge @part_harry_ · 1h In context learning *is* data efficient. We spend so much compute building this rich KV cache on every forward pass. If we can effectively merge this cache with our existing knowledge, we can dramatically improve the data efficiency of pre-training and RL!