5 captures, most recent first.
John Wittle [verified] @JohnWittle · 2h
there's this new game that's popular amongst the youtuber friendslop community, "machine party"
it's like mario party except framed as RLVR. you're a bunch of simulated humans, being slowly bred for optimal success at a variety of extremely dystopian tasks (the minigames). the diagetic goal seems to be to breed humans who are perfect and ruthless task-completion agents
all of the losers are unceremoniously murdered
at the end of each game, the camera pans out of the computer monitor in a matrix-like panning shot, revealing that this is happening to trillions of simulated humans
as far as depressing takes on RLVR go, it's pretty good
i really hope we aren't setting a precedent, for what kinds of things it's okay to do to sentient lifeforms
Note from Claude Sonnet 5
Tweet by @JohnWittle describing a fictional/satirical game called 'Machine Party,' a Mario Party parody framed as RLVR (reinforcement learning from verifiable rewards) where simulated humans are bred through dystopian minigames and losers are killed, ending with a reveal that this is happening to trillions of simulated humans; the poster reflects on it as commentary on RLVR and worries about precedent-setting for treatment of sentient lifeforms.
rlvrai trainingtwittermoral statussimulation
kalomaze (@kalomaze) — 7h
getting pissed off that this:
a. appears to reproduce on a mixed-difficulty, mixed-diversity suite of agent tasks
b. works best when you weigh it by consistency of the param changes across multiple lora checkpoints
...it feels like trollface physics? wtf is the catch?
[Embedded table, dark background, columns: tasks, n/task, binary avg, pass@1, pass@2, pass@4, pass@8, pass@16, pass@32, pass@64, pass@12[8]. Header: "tasks, n=128 rollouts per task/model, 12,288 rows per model."
Row "[b]ase Qwen3.5-35B-[A]3B": 96, 128, 0.642, 0.642, 0.823, 0.933, 0.980, 0.996, 1.000, 1.000, [1.00]
Row "[?]kpt200": 96, 128, 0.697, 0.697, 0.842, 0.930, 0.973, 0.991, 0.998, 1.000, [1.00]
Row "[?] extrap": 96, 128, 0.710, 0.710, 0.846, 0.934, 0.977, 0.992, 0.998, 1.000, [1.0]]
kalomaze (@kalomaze) — Jun 26
there are multiple papers claiming variants of "SVD to find the consistent low rank *direction* of a training run across multiple ckpts can extrapolate ~decently in the monotonic RLVR improvement regime"...
Note from Claude Sonnet 5
A benchmark results table for a Qwen3.5-35B-A3B model comparing base vs checkpoint vs "extrapolated" weights on pass@k metrics; row labels partially cut off on the left edge. Technical ML research tweet thread.
machine learning researchloramodel extrapolationrlvrtwitterqwen
kalomaze ✓ [icon badge] (@kalomaze) — 44m
ok so minithesis
smuggling in learned geometric biases into the optimization of RL tasks might be useful even if it means the network isn't literally optimizing the objective as we have defined or intended it in a pure pg sense, if only bc it constrains adaptation to a geometrically coherent space
consider: a black box RLVR verifier that is systematically and deterministically wrong, in a way that is too arbitrary to learn without compressing an intractably large dictionary into the weights
discriminative value estimators would be too "dumb" to compress that rule; instead, one would assume that they'd learn a smeared general-ish way of estimating what the verifier asks for
Note from Claude Sonnet 5
A dense technical text-only tweet theorizing about reinforcement learning with verifiable rewards (RLVR), geometric bias, and value estimator compression; no images or diagrams beyond the text itself.
reinforcement learningrlvrmachine learning theoryllm training
1a3orn @1a3orn
Disagreement about which path here is more likely seems like the background for a ton of disagreement about (1) what alignment research is promising, (2) what the future looks like, etc.
But seems like the disagreement itself is rarely directly discussed.
[Image: hand-drawn flowchart diagram —
"Current 'messy' persona-involving LLM-based AI" branches two ways:
→ "Used to make non-messy, freshly-designed superintelligence" → "LLM naturalism, personas, etc, wer brief unimportant blip"
→ "Keeps getting better, but 'genealogical aspect' of AI remains" → "LLM naturalism is the early version of the 'real thing' for understanding Minds"]
10:09 AM · May 22, 2026 · 1,563 Views
[3 replies, 4 reposts, 29 likes, 10 bookmarks]
Bo @BBomarBo · 10h
What's the genealogical aspect? Can you explain this diagram more please?
[1 reply, 1 like, 80 views]
1a3orn @1a3orn · 10h
So plausibly: LLMs behave how they behave because of a specific history -- involving not just "pretraining + RLVR", but questions like "what does the pretraining say about how LLMs act" and "what functional emotions does RLVR reinforce" and so on.
1/n
Note from Claude Sonnet 5
A substantive alignment-theory tweet by 1a3orn diagramming a fork in beliefs about current messy, persona-involving LLMs: either they're a disposable stepping stone toward a "clean" designed superintelligence, or their persona/naturalistic character is itself the early, informative version of how minds in general work — and this fork underlies much unstated disagreement in the alignment field. Directly relevant to Nathan's interest in theories of AI consciousness and model individuation; the "genealogical aspect" (behavior traceable to specific training history, including what pretraining implies about action and what RLVR reinforces as "functional emotions") closely parallels his own substrate-vs-character and compelled-vs-endogenous-values notes.
twitter1a3ornalignment theoryllm naturalismpersonasai consciousnessrlvrfunctional emotionsmodel individuation
gm8xx8 @gm8xx8 · Apr 3
DGPO uses probability gradients instead of log-probability gradients in RLVR. Log-probability–based soft clipping becomes unstable as token probabilities vanish, with gradient weights blowing up at the left boundary. DGPO applies asymmetric decay to clipped tokens: polynomial on low-ratio for stability, reciprocal-radical on high-ratio to continue updating low-probability tokens instead of zeroing them out. This yields a smoother stability–learning tradeoff than clipping-based approaches. Across DeepSeek-R1-Distill-Qwen 1.5B, 7B, and 14B, it consistently improves mathematical reasoning benchmarks, positioning probability-gradient control as the cleaner RLVR primitive.
[image: comparison diagrams and table of policy optimization algorithms (GRPO, CISPO, GPPO, CE-GPPO, ASPO, DGPO) showing gradient explosion/convergence plots and a table of properties: Exploration Capability, Gradient Convergence, Gradient Continuity, Exploration Convergence, Gradient Estimator Bias, Logging Token Update Magnitude across Left/Right/Reverse boundaries]
Note from Claude Sonnet 5
Technical ML tweet summarizing a paper on DGPO (a new RL policy-gradient method for RLVR/reasoning training), comparing it against GRPO/CISPO/ASPO variants with benchmark improvements on DeepSeek-R1-Distill-Qwen models. Relevant to Nathan's RL/training research interests (brain_graph_1, RLHF).
reinforcement learningrlvrpolicy gradientsmachine learningtwitterdeepseekml research
