sparsity — Timeline

2025-07-11

kalomaze @kalomaze · Jul 2 lr=1.0, clip=1e-10 → 99.13% of parameters unchanged compared to the base model. (orange run) lr=0.01, clip=1e-8 → 95.21% of parameters unchanged (purple run) both converge to very similar points, except one is significantly more sparse in what it chooses to update [two charts: "train/loss" and "eval/loss" for "coherence-classifier-v1-18k-both" runs, both converging to ~0.1 by step 300] kalomaze @kalomaze · Jul 2 Replying to @kalomaze >learning_rate: 1e-2 >max_grad_norm: 0.0000001 no really, this works.

Note from Claude Sonnet 5

Continuation of kalomaze's technical ML training thread — extreme gradient-clipping experiments producing very sparse parameter updates while still converging to similar loss. General ML training technique discussion, not AI-safety focused.

machine-learning training gradient-clipping sparsity optimization technical

rohan anil @_arohan_

2025-05-10 — quoting @liang_we... (Weixin Li)

rohan anil @_arohan_ · 4h This is really cool work! I wonder if we could generalize even better by introducing modality as feature embedding to the router instead. That is router gets privileged information. > QUOTED: Weixin Li... @liang_we... · May 9 > 🎉 Excited to share: "Mixture-of-Transformers (MoT)" has been officially accepted to TMLR (March 2025) and the code is now open-sourced! > ... > Show more [Embedded images: paper figures — "Mixture-of-Transformer: Sparsity for all non-embedding..." formula showing modality-aware routing (W_image, W_text, W_speech), and a training-loss chart comparing Dense Transformer, MoE-4x, and MoT (ours) showing MoT achieving ~2x speedup / efficiency gains via modality-aware sparsity, training a 7B model from scratch.]

Note from Claude Sonnet 5

A technical ML Twitter exchange about the "Mixture-of-Transformers" (MoT) paper (accepted TMLR March 2025), a modality-aware sparse architecture showing ~2x training efficiency gains over dense transformers, with a follow-up suggestion to route based on modality embeddings. General ML architecture research, not directly safety-related but relevant to Nathan's technical tracking of model architecture trends (adjacent to his own brain_graph_1 work on routing/sparsity).

twitter machine-learning mixture-of-transformers model-architecture sparsity moe rohan-anil