machine-learning

Plain text reply thread, no images; technical ML explainer from Grok.

Plain text tweet; avatar photo of a woman in sunglasses.

Text-only post, cut off at bottom edge (engagement icons partially visible but counts not legible); no images.

A practitioner's tweet on training small (~1B parameter) language models, comparing SFT, DPO, and KTO fine-tuning methods and noting that chaining them in sequence acts as a regularizer against overfitting. Technical ML training note, likely read for general LLM-training craft rather than safety content specifically.

A technical ML tweet describing "Cautious Weight Decay" (CWD), a simple one-line modification to AdamW-style decoupled weight decay that only applies decay where the sign matches the optimizer update, validated in the Modded NanoGPT speedrun community. Relevant to Nathan's own ML/optimizer engineering work on brain_graph_1 (which is already tracking Muon and other optimizer choices per project memory) — a candidate technique worth evaluating for that project.

A tweet on an architectural trick — training a network to predict the gradient of the loss (d_loss/d_y) and applying iterative gradient descent at inference, framed as "gradient boosting with a single shared learner" — quoting an older tweet framing diffusion models as adaptive computation time. Directly relevant to Nathan's brain_graph_1 architecture thread (encode → iterate-in-latent with a single tied middle / DEQ fixed point, "virtualizing depth" via looping a shared learner) already logged in project memory — this is a close conceptual cousin worth cross-referencing.

A technical ML-engineering thread on memory optimization for training with large vocabularies — fusing linear readout with cross-entropy loss to avoid materializing the full logits matrix, referencing Apple's ml-cross-entropy library. Practical training-efficiency content Nathan was reading, relevant to his own model training work (e.g. brain_graph_1).

A technical Twitter thread between ML researchers Jifan Zhang and Dimitris Papailiopoulos debating curriculum/problem-generation strategies for RL training on math reasoning (decomposing math papers into subproblems, IMO-question generation vs solving difficulty). ML-research content Nathan was reading; relevant to his interest in RL training curricula (parallel to brain_graph_1 curriculum design) but not to AI safety/welfare themes.

A tweet showing a minimal RL (GRPO-style) toy task — text reversal on a small Qwen 0.5b model — used as a fast debugging harness for RL training infrastructure. Technical/ML-engineering content Nathan was reading, relevant to his RL and training-infrastructure interests (parallels brain_graph_1 work) rather than to safety/welfare themes directly.

Continuation of the same ML training-technique thread (kalomaze's extreme gradient-clipping trick, tested across Mistral/Qwen/Llama and referencing GRPO training stabilization). Technical ML discussion, not AI-safety focused. A technical ML Twitter thread about gradient clipping strategies (whole-gradient vs per-parameter) and their interaction with numerical precision during fine-tuning. Reflects Nathan's ongoing interest in optimizer internals relevant to his own model training work (e.g. brain_graph_1).

Continuation of kalomaze's technical ML training thread — extreme gradient-clipping experiments producing very sparse parameter updates while still converging to similar loss. General ML training technique discussion, not AI-safety focused.

Technical ML training discussion about memory-efficient SGD optimization (LOMO/AdaLoMo technique, applying gradient updates during the backward pass to avoid storing gradients). Not AI-safety focused, general ML training tip.

Continuation of the same ML training-technique thread as the prior screenshot (kalomaze discussing pairwise/contrastive classification loss formulation). Technical ML discussion, not AI-safety focused.

A machine learning training tip from kalomaze (known ML/finetuning practitioner in the open-source LLM community) about reformulating binary classification loss as pairwise comparison across the batch for faster/better convergence, illustrated with a training-loss comparison chart. Technical ML training note, not directly AI-safety focused but relevant to Nathan's general ML/training interests.

A technical ML tweet about a PyTorch memory-optimization trick (backward-passing multiple disentangled losses separately instead of summing them first) applied to Google's TIPS multi-loss image/text encoder architecture, cutting GPU memory 3x. Relevant to Nathan's own ML engineering work (e.g. brain_graph_1 training on constrained GPU budgets); a practical technique worth potentially reusing.

A tweet thread from ML researcher Zeyuan Allen-Zhu (known for the "Physics of Language Models" paper series) describing scrappy compute-scavenging tactics (idle-GPU sniffing across other teams' clusters, working nights/weekends) used during resource-constrained research. Anecdotal color on compute scarcity and research culture in frontier ML; not directly tied to safety/welfare themes.

A technical ML Twitter exchange about the "Mixture-of-Transformers" (MoT) paper (accepted TMLR March 2025), a modality-aware sparse architecture showing ~2x training efficiency gains over dense transformers, with a follow-up suggestion to route based on modality embeddings. General ML architecture research, not directly safety-related but relevant to Nathan's technical tracking of model architecture trends (adjacent to his own brain_graph_1 work on routing/sparsity).

A technical ML Twitter thread arguing that KTO (Kahneman-Tversky Optimization) outperforms DPO (Direct Preference Optimization) variants for offline RL/post-training in controlled settings, with a claim that DPO-variant papers are more prone to overstated/unreliable results than what practitioners actually use internally. Relevant to Nathan's RLHF/post-training research interests.

A technical Twitter debate disputing Ilya Sutskever's thesis on language modeling as approximating an "adult mind," arguing instead that current LLM training via next-token prediction on human text is behavior cloning (crystallized skill) rather than reproducing a child's experiential learning capacity. Relevant to Nathan's interest in AI cognitive architecture and learning-paradigm debates. Continuation of the same Twitter thread as the previous screenshot — invoking a famous Alan Turing quote (from his 1950 "Computing Machinery and Intelligence" paper) about simulating a child's mind and educating it, as historical grounding for the "child-mind not adult-mind" critique of current LLM training. Same thread as Screenshot_20250420-105319. Continuation of the same thread (see Screenshot_20250420-105319/105333) — argues that text is a "sparse, biased projection" of humanity's world model, that pretraining lacks the "direct friction" of embodied childhood learning, and that superhuman capability requires self-supervised world-model generation rather than human-text imitation. Substantive argument about limits of LLM pretraining vs. embodied/self-supervised learning.

A machine-learning Twitter post complaining about researchers who inflate paper titles/rename existing methods for credit, contrasted with well-known "GOAT" papers that undersell themselves with plain titles. General ML culture commentary, not directly about AI safety.