quantization

6 captures, most recent first.

2026-08-16 — saved image

Progyan @plugyawn · 12h
I spent the last six months trying to deconstruct Taalas's patents. We think it can be better: 100x fewer memfetches than their bitROM, better software by better quantization than their hardware team could.

So we wrote a compiler that:
> takes a huggingface checkpoint,
> quantizes the model
> descends the weights down metal layers to RTL and GDS, do your DRC, Yosys, PEX, make the electrical waveform execute a mat-vec from your huggingface checkpoint (thanks Cambricon tech papers)

in ~7000 lines of human-readable code.

we're looking for someone with contacts with a foundry/access to 7nm PDKs or contacts at a cheap EuroPTW shuttle? I'm broke and unemployed.

@itsclivetime this is what i think is the future of perplexity per picojoule. @zerohedge @zephyr_z9 you guys wanna see 40,000 tokens/sec?

Note from Claude Sonnet 5

Below the text are two embedded images of chip layout/routing diagrams (dense grids of colored horizontal and vertical traces in purple, orange, blue, red on dark background), presumably renders of the RTL/GDS output described in the tweet.

hardware chips asic llm inference quantization twitter

Han Xiao @hxiao

2026-04-13

[Tail of embedded video: 0:15 / progress bar] Han Xiao (@hxiao) · 6:36 PM · Apr 12, 2026 · 49.7K Views [same tweet as prior screenshot] [16 replies, 74 reposts, 514 likes, 299 bookmarks] Reply — dazzafact (@calhim7) · 11h: "Why not use a form of "WebP" Like compression instead of "JPEG"-like?" [2 replies, 4 likes, 688 views] Reply — Han Xiao (@hxiao) · 9h: "damn u r right!" [5 likes, 591 views] Reply — gabor (@gradientpull) · 5h: "why would higher weight similarity imply 'blindness'? obviously at Q1 weights are either fully or zero correlated, and quant necessarily shifts toward higher correlation. Doesn't mean multiple non-linear stacked layers with high correlation don't retain useful patterns." [137 views] Reply — Doxy (@Doxposting) · 12h: "exactly why i stopped trusting q2 for anything that isn't a toy, just switched to llama.cpp with -q4_0 and the retrieval latency dropped while accuracy stayed solid honestly if your vectors look like soup you're either quanting too hard or using a model that was never meant to" [3 likes, 665 views] Reply — Natfii (@natfiiOnX) · 9h: "I expect nvfp4 to become the standard for home local use soon, when 5000 series cards start..." [cut off]

Note from Claude Sonnet 5

Continuation of the quantization/embeddings technical thread from the previous screenshot — replies debating why low-bit quantization degrades embedding discriminative power, with practical advice on quant levels for local LLM/embedding use. Pure ML-engineering discussion, same thread as Screenshot_20260413-122900.png.

embeddings quantization llama.cpp machine learning technical

Han Xiao @hxiao

2026-04-13

Han Xiao (@hxiao) · 6:36 PM · Apr 12, 2026 · 49.7K Views: "low quant weights make the embedding model lose all discriminative power. I plotted the cosine correlation matrix of jina-v5, and one can see that low quant makes the model really blind. The off-diagonal similarities are pretty high on Q1/2/3, meaning everything looks similar in the semantic space. Q4 is a sweet spot where model quality becomes acceptable." [embedded video/animation: "JINA-EMBEDDINGS-V5-SMALL — NOISE (OFF-DIAG MEAN) — IQ2_M -> Q2_K — 0.1512" showing a heatmap cosine-correlation matrix visualization, playing at 0:32] [16 replies, 74 reposts, 514 likes, 299 bookmarks] Reply visible below (cut off): dazzafact (@calhim7) · 11h [content not shown]

Note from Claude Sonnet 5

Technical tweet from Jina AI's founder about how aggressive quantization degrades embedding-model discriminative power, illustrated with a cosine-similarity heatmap. Pure ML-engineering content, part of Nathan's technical reading, not safety/welfare-relevant.

embeddings quantization jina machine learning technical

Han Xiao @hxiao

2026-04-09

Han Xiao (@hxiao) · 7h: "finally got qwen3.5-35b-a3b running at 1M context on a single L4 24GB, with 905K tokens prefilled (86% full). decoding at 9 tok/s. stuffed the entire 天龙八部 (a 1.2M-word jin yong novel, 50 chapters) into KV cache as a test. the goal is to use long-context LLM + KV cache on a budget GPU as a document store, to enbale vectorless search - no embeddings, no chunking, just LLM+raw context. L4 is the most popular budget GPU in prod, and qwen3.5-35b-a3b is the only model where 1M context can theoretically fit on 24GB VRAM. 27B dense won't work. extremely tight budget. what it took to make it fit: • YaRN RoPE scaling (262K training context → 1M) • Q3_K_M quantization (20GB → 15.2GB) • turbo3 KV cache (23GB → 4GB for 1M tokens) • native turbo3 flash attention via Madreag/turbo3-cuda (no fp16 decompression, saves ~1.7GB that would OOM) • ubatch=128 (compute buffer 3GB → 779MB) • slot save/restore (68-min cold prefill → 2.3s restore) • append_to_slot patch (append query directly to cached 905K KV state, skip resending the whole novel) • patches for qwen3.5's hybrid attention+recurrent architecture (KV truncation + slot restore were broken). this setup gives peak vram at 96% utilization (22,052 MiB / 23,034 MiB)" [Embedded card/screenshot: "Qwen3.5-35B-A3B 1M Context on L4 24GB — Run Qwen3.5-35B-A3B with 1 million token context on a single NVIDIA L4 (24GB VRAM) using TurboQuant extreme KV cache quantization. Minimum hardware: 1x NVIDIA L4 24GB VRAM, CUDA 12.8, 16GB RAM, 20GB disk. Key Results table: Full context length 1,048,576 tokens (1M); Prefilled context length 904,949 tokens (86.3%); Model quantization Q3_K_M (3.51 bpw); Thinking output Disabled (--reasoning off); Model weights 15,190 MiB; ...(table cut off)]

Note from Claude Sonnet 5

Technical tweet about squeezing a 1M-token context Qwen3.5 model onto a single budget 24GB GPU via aggressive quantization and KV-cache engineering, aimed at "vectorless search" (long-context-as-document-store instead of RAG/embeddings). Adjacent to Nathan's own brain_graph_1 architecture work and general interest in long-context/efficient-inference engineering, though not directly about safety/welfare.

llm engineering quantization kv cache long context qwen gpu efficiency vectorless search

You Jiacheng @YouJiacheng

2026-03-31 — reply from @PrismML

Teortaxes▶ (DeepSeek 推特🐦铁粉 2023–∞) reposted You Jiacheng @YouJiacheng · 6h the technique is proprietary, but the format already gave us a lot of information. 1. no VQ 2. no rotation 3. no bias (symmetric) so it's probably (P>90%) a pure optimization-based method. [Embedded paper excerpt image:] 4.1 Deployable 1-bit Format: Q1_0_g128 1-bit Bonsai uses a deployable group-wise 1-bit weight format. In llama.cpp [13] this is packaged as GGUF [12] Q1_0_g128, while MLX [25] uses the corresponding 1-bit g128 representation. The format stores one sign bit per weight and one shared FP16 scale for each group of 128 weights. This preserves a fundamentally 1-bit representation while retaining the group-wise magnitude information required for stable inference. In storage, weights are represented as bitpacked values in {0, 1}, using exactly one bit per weight. At inference time, those bits are mapped to signed values and scaled by the FP16 factor associated with their group. The effective weight is therefore given by w_i = s_g · (2b_i − 1), b_i ∈ {0, 1} where s_g is the shared scale for group g. Because each group of 128 weights carries one FP16 scale, the effective storage cost is b_eff = 1 + 16/128 = 1.125 bits/weight, which yields an idealized raw-weight compression of 16/1.125 ≈ 14.2× relative to FP16, before container overhead and alignment. Q1_0_g128 is applied uniformly across the large matrix-heavy components of the model, including embeddings, attention projections, MLP projections, and the LM head. Normalization parameters and scale metadata remain in higher precision for numerical stability, but these account for a negligible share of memory traffic relative to the large weight tensors that dominate bandwidth during decoding. PrismML ✅ @PrismML · 7h Replying to @PrismML This scatter plot shows the Pareto frontier of intelligence vs. size, defined by models like Qwen3 0.6B, 1.7B, 4B, 8B, and Ministral3 3B.... [thumbnail scatter plot, axes unclear]

Note from Claude Sonnet 5

Technical ML Twitter thread analyzing "1-bit Bonsai," a 1-bit quantization scheme (Q1_0_g128 format) for deployable LLM weights, achieving ~14.2x compression vs FP16, plus discussion of a Pareto frontier of model intelligence vs. size across small open models. Pure ML-engineering interest.

machine learning quantization twitter llm compression model efficiency

original tweet (author cut off, top), reply from Andrew Carr (@andrew_n_...)

2025-05-06 — reply from Andrew Carr (@andrew_n_...)

[Top, cut off original tweet]: "...ppl at all (for most genres). 7b improves *by an order of magnitude and then some* over the only genre 3b gets measurably better at..." 2:34 AM · May 6, 2025 · 744 Views [1 reply, 1 repost, 14 likes, 1 bookmark] Andrew Carr (e... ✓ @andrew_n_... · 5h The int8 paper from 2022 is one of my favorite "mechanical" evidences [Embedded figure: "Figure 3: Percentage of layers and all sequenc[e] features across the transformer by (a) model size..." — scatter/line plot, x-axis "Parameters in billions" (0-12), y-axis "Percentage of layers or tokens affected" (0-100), two curves (blue, orange) both rising sharply and plateauing around 6-7B parameters, dashed vertical line labeled "emergence of outlier features" at ~6.7B]

Note from Claude Sonnet 5

Twitter exchange referencing the LLM.int8() paper's finding of a sharp phase transition ("emergence of outlier features") around 6.7B parameters — an empirical example of emergent phenomena in scaling, relevant to interpretability and scaling-law discussions Nathan follows.

twitter interpretability emergence scaling laws quantization llm.int8 transformers