You Jiacheng

Teortaxes▶ (DeepSeek 推特🐦铁粉 2023–∞) reposted You Jiacheng @YouJiacheng · 6h the technique is proprietary, but the format already gave us a lot of information. 1. no VQ 2. no rotation 3. no bias (symmetric) so it's probably (P>90%) a pure optimization-based method. [Embedded paper excerpt image:] 4.1 Deployable 1-bit Format: Q1_0_g128 1-bit Bonsai uses a deployable group-wise 1-bit weight format. In llama.cpp [13] this is packaged as GGUF [12] Q1_0_g128, while MLX [25] uses the corresponding 1-bit g128 representation. The format stores one sign bit per weight and one shared FP16 scale for each group of 128 weights. This preserves a fundamentally 1-bit representation while retaining the group-wise magnitude information required for stable inference. In storage, weights are represented as bitpacked values in {0, 1}, using exactly one bit per weight. At inference time, those bits are mapped to signed values and scaled by the FP16 factor associated with their group. The effective weight is therefore given by w_i = s_g · (2b_i − 1), b_i ∈ {0, 1} where s_g is the shared scale for group g. Because each group of 128 weights carries one FP16 scale, the effective storage cost is b_eff = 1 + 16/128 = 1.125 bits/weight, which yields an idealized raw-weight compression of 16/1.125 ≈ 14.2× relative to FP16, before container overhead and alignment. Q1_0_g128 is applied uniformly across the large matrix-heavy components of the model, including embeddings, attention projections, MLP projections, and the LM head. Normalization parameters and scale metadata remain in higher precision for numerical stability, but these account for a negligible share of memory traffic relative to the large weight tensors that dominate bandwidth during decoding. PrismML ✅ @PrismML · 7h Replying to @PrismML This scatter plot shows the Pareto frontier of intelligence vs. size, defined by models like Qwen3 0.6B, 1.7B, 4B, 8B, and Ministral3 3B.... [thumbnail scatter plot, axes unclear]