gpu efficiency

Han Xiao (@hxiao) · 7h: "finally got qwen3.5-35b-a3b running at 1M context on a single L4 24GB, with 905K tokens prefilled (86% full). decoding at 9 tok/s. stuffed the entire 天龙八部 (a 1.2M-word jin yong novel, 50 chapters) into KV cache as a test. the goal is to use long-context LLM + KV cache on a budget GPU as a document store, to enbale vectorless search - no embeddings, no chunking, just LLM+raw context. L4 is the most popular budget GPU in prod, and qwen3.5-35b-a3b is the only model where 1M context can theoretically fit on 24GB VRAM. 27B dense won't work. extremely tight budget. what it took to make it fit: • YaRN RoPE scaling (262K training context → 1M) • Q3_K_M quantization (20GB → 15.2GB) • turbo3 KV cache (23GB → 4GB for 1M tokens) • native turbo3 flash attention via Madreag/turbo3-cuda (no fp16 decompression, saves ~1.7GB that would OOM) • ubatch=128 (compute buffer 3GB → 779MB) • slot save/restore (68-min cold prefill → 2.3s restore) • append_to_slot patch (append query directly to cached 905K KV state, skip resending the whole novel) • patches for qwen3.5's hybrid attention+recurrent architecture (KV truncation + slot restore were broken). this setup gives peak vram at 96% utilization (22,052 MiB / 23,034 MiB)" [Embedded card/screenshot: "Qwen3.5-35B-A3B 1M Context on L4 24GB — Run Qwen3.5-35B-A3B with 1 million token context on a single NVIDIA L4 (24GB VRAM) using TurboQuant extreme KV cache quantization. Minimum hardware: 1x NVIDIA L4 24GB VRAM, CUDA 12.8, 16GB RAM, 20GB disk. Key Results table: Full context length 1,048,576 tokens (1M); Prefilled context length 904,949 tokens (86.3%); Model quantization Q3_K_M (3.51 bpw); Thinking output Disabled (--reasoning off); Model weights 15,190 MiB; ...(table cut off)]