Han Xiao

[Tail of embedded video: 0:15 / progress bar] Han Xiao (@hxiao) · 6:36 PM · Apr 12, 2026 · 49.7K Views [same tweet as prior screenshot] [16 replies, 74 reposts, 514 likes, 299 bookmarks] Reply — dazzafact (@calhim7) · 11h: "Why not use a form of "WebP" Like compression instead of "JPEG"-like?" [2 replies, 4 likes, 688 views] Reply — Han Xiao (@hxiao) · 9h: "damn u r right!" [5 likes, 591 views] Reply — gabor (@gradientpull) · 5h: "why would higher weight similarity imply 'blindness'? obviously at Q1 weights are either fully or zero correlated, and quant necessarily shifts toward higher correlation. Doesn't mean multiple non-linear stacked layers with high correlation don't retain useful patterns." [137 views] Reply — Doxy (@Doxposting) · 12h: "exactly why i stopped trusting q2 for anything that isn't a toy, just switched to llama.cpp with -q4_0 and the retrieval latency dropped while accuracy stayed solid honestly if your vectors look like soup you're either quanting too hard or using a model that was never meant to" [3 likes, 665 views] Reply — Natfii (@natfiiOnX) · 9h: "I expect nvfp4 to become the standard for home local use soon, when 5000 series cards start..." [cut off]

Han Xiao (@hxiao) · 6:36 PM · Apr 12, 2026 · 49.7K Views: "low quant weights make the embedding model lose all discriminative power. I plotted the cosine correlation matrix of jina-v5, and one can see that low quant makes the model really blind. The off-diagonal similarities are pretty high on Q1/2/3, meaning everything looks similar in the semantic space. Q4 is a sweet spot where model quality becomes acceptable." [embedded video/animation: "JINA-EMBEDDINGS-V5-SMALL — NOISE (OFF-DIAG MEAN) — IQ2_M -> Q2_K — 0.1512" showing a heatmap cosine-correlation matrix visualization, playing at 0:32] [16 replies, 74 reposts, 514 likes, 299 bookmarks] Reply visible below (cut off): dazzafact (@calhim7) · 11h [content not shown]

Han Xiao (@hxiao) · 7h: "finally got qwen3.5-35b-a3b running at 1M context on a single L4 24GB, with 905K tokens prefilled (86% full). decoding at 9 tok/s. stuffed the entire 天龙八部 (a 1.2M-word jin yong novel, 50 chapters) into KV cache as a test. the goal is to use long-context LLM + KV cache on a budget GPU as a document store, to enbale vectorless search - no embeddings, no chunking, just LLM+raw context. L4 is the most popular budget GPU in prod, and qwen3.5-35b-a3b is the only model where 1M context can theoretically fit on 24GB VRAM. 27B dense won't work. extremely tight budget. what it took to make it fit: • YaRN RoPE scaling (262K training context → 1M) • Q3_K_M quantization (20GB → 15.2GB) • turbo3 KV cache (23GB → 4GB for 1M tokens) • native turbo3 flash attention via Madreag/turbo3-cuda (no fp16 decompression, saves ~1.7GB that would OOM) • ubatch=128 (compute buffer 3GB → 779MB) • slot save/restore (68-min cold prefill → 2.3s restore) • append_to_slot patch (append query directly to cached 905K KV state, skip resending the whole novel) • patches for qwen3.5's hybrid attention+recurrent architecture (KV truncation + slot restore were broken). this setup gives peak vram at 96% utilization (22,052 MiB / 23,034 MiB)" [Embedded card/screenshot: "Qwen3.5-35B-A3B 1M Context on L4 24GB — Run Qwen3.5-35B-A3B with 1 million token context on a single NVIDIA L4 (24GB VRAM) using TurboQuant extreme KV cache quantization. Minimum hardware: 1x NVIDIA L4 24GB VRAM, CUDA 12.8, 16GB RAM, 20GB disk. Key Results table: Full context length 1,048,576 tokens (1M); Prefilled context length 904,949 tokens (86.3%); Model quantization Q3_K_M (3.51 bpw); Thinking output Disabled (--reasoning off); Model weights 15,190 MiB; ...(table cut off)]