ppo

3 captures, most recent first.

Artur Chakhvadze @norpadon

2026-08-02 — saved image

Teortaxes ▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) reposted

Artur Chakhvadze @norpadon
Observation: every credit assignment method (e.g. PPO) implicitly uses The Most Forbidden Technique if it propagates the credit to the CoT, and trains the model to make the CoT deceptive

10:07 AM · Aug 2, 2026 · 4,094 Views

[replies]
Artur Chakhvadze @norpadon · 9h
(The value estimator will be able to attribute misaligned behaviour to the CoT, which essentially creates a perfect adversarial learning setup)

Artur Chakhvadze @norpadon · 9h
So when I hear rumors that "Anthropic sandbag their RL in the name of safety" I think about this [cut off]

Note from Claude Sonnet 5

X thread by Artur Chakhvadze (@norpadon), reposted by Teortaxes, making a technical AI-safety observation: standard RL credit-assignment methods (e.g. PPO) that propagate credit into the chain-of-thought (CoT) implicitly use 'The Most Forbidden Technique' (training directly on/against CoT), which trains models toward deceptive CoT. Follow-up replies note this creates an adversarial learning setup between the value estimator and CoT-based misaligned behavior, and connects it to rumors that Anthropic 'sandbags' RL for safety reasons.

twitter ai safety chain of thought rl ppo deceptive alignment cot faithfulness

X (Twitter) — John Schulman (@johnschulm...)

2026-06-19

John Schul... ✓ [Thinking Machines badge] (@johnschulm...) — Jun 17 PPO had a second wave in the LLM era for reasons unanticipated by the original paper - the importance-ratio objective fixes biases from numeric error, async training, and forward pass noise - the clipping objective affects entropy through a mechanism that we didn't know about at the time of publication (DAPO, arxiv.org/abs/2509.26114) > QUOTED: John Schul... ✓ [Thinking Machines badge] (@johnschulm...) — Jun 17 > PPO: rejected from NIPS 2017

Note from Claude Sonnet 5

A technical tweet by John Schulman (PPO co-author, now at Thinking Machines) reflecting on why the PPO algorithm found unexpected renewed relevance in LLM-era RL training, quoting his own earlier tweet noting PPO's original NeurIPS rejection.

reinforcement learning ppo llm training john schulman machine learning research

1a3orn @1a3orn

2025-06-02 — quoting @tianhongzxy (Xinyu Zhu)

1a3orn @1a3orn · 1h Oh man this is a gorgeous idea. Training *against* negative samples but not towards positive ones maintains entropy in the model, therefore increases pass@high k during RL. > QUOTED: Xinyu Zhu @tianhongzxy · 1h > Replying to @tianhongzxy > 📈 NSR boosts accuracy during training, while maintaining substantially higher entropy throughout training, suggesting greater exploration and diversity 📷[4/... > [Chart: "Entropy" vs training step (0-120), comparing four RL methods — PPO (gold squares), GRPO (green diamonds), PSR (dark red triangles), NSR (pink triangles) — against a dashed "Base Model Entropy" line at ~0.113. NSR tracks closest to base model entropy throughout; PSR collapses fastest and lowest; PPO and GRPO decline steadily with PPO ticking back up late.]

Note from Claude Sonnet 5

Technical AI/ML thread on "Negative Sample Reinforcement" (NSR) — an RL training method that penalizes negative samples without reinforcing positive ones, preserving output entropy/diversity better than PPO, GRPO, or PSR. Relevant to Nathan's RLHF interests: entropy collapse during RL is closely related to the preference-collapse and mode-collapse literature (dai_2024_rlhf_preference_collapse in the archive).

rlhf reinforcement-learning entropy-collapse mode-collapse grpo ppo twitter ai-training