reinforcement-learning

Jifan Zhang @jifan_zhang · 3h you could decompose math papers into smaller subproblems. they did claim it's a general model. 💬 1 🔁 ❤ 1 📊 203 ↗ Dimitris Papailio... @Dimitris... · 2h I don't understand. All I'm saying is that you need a curriculum type of problem description generation during RL 💬 1 🔁 ❤ 4 📊 210 ↗ Jifan Zhang @jifan_zhang · 2h i agree getting a curriculum is necessary and probably easy once you have the questions. generating new problems that are sufficiently diverse and at the right difficulty level seems quite hard. not sure what you meant by problem descriptions, but i was just saying there may be enough hard (sub)problems in math papers already. 💬 1 🔁 ❤ 1 📊 110 ↗ Jifan Zhang @jifan_zhang · 1h fwiw, creating IMO questions is generally considered much harder than solving them, but they also require somewhat different skills. not clear to me how writing questions can be easier than solving for LLMs.

kalomaze @kalomaze · 1m prime-rl has a cute toy task for text reversal on Qwen 0.5b, which is used to debug simpler things (i.e. custom wandb logging, implementation correctness). it uses 2 GPUs total and takes less than 5 minutes to finish. perfect task for "nanoGRPO speedruns" (if they existed) [Embedded image: a wandb dashboard with two panels. Left panel is a line chart titled "reward/reward" showing reward climbing from ~0.2 to plateau around 0.75-0.8 over ~25 training steps. Right panel is a table titled "runs.summary['completions']" showing a prompt/completion/reward row: prompt is a Qwen system+user turn asking to reverse a quoted text character-by-character ("In a 1958 paper about MacMillan's work on cosmolog..."), completion shows a reversed-text answer in <answer> tags, reward 0.8932.]

1a3orn @1a3orn · 1h Oh man this is a gorgeous idea. Training *against* negative samples but not towards positive ones maintains entropy in the model, therefore increases pass@high k during RL. > QUOTED: Xinyu Zhu @tianhongzxy · 1h > Replying to @tianhongzxy > 📈 NSR boosts accuracy during training, while maintaining substantially higher entropy throughout training, suggesting greater exploration and diversity 📷[4/... > [Chart: "Entropy" vs training step (0-120), comparing four RL methods — PPO (gold squares), GRPO (green diamonds), PSR (dark red triangles), NSR (pink triangles) — against a dashed "Base Model Entropy" line at ~0.113. NSR tracks closest to base model entropy throughout; PSR collapses fastest and lowest; PPO and GRPO decline steadily with PPO ticking back up late.]

``` thebes @voooooogel · 23m Coming back to this after the yak-shave of all yak-shaves building logitloom with some interesting findings. 1. R1 thinking traces are INCREDIBLY diverse. I ran a depth 10, top P 95% tree, and after having to stop expanding it early for fear of crashing my VLLM instance under load, it had discovered >2,500 leaf tokens! (Some nodes are folded in the above screenshot, which is why it may look like <10 tokens.) Given that I stopped it while it was still expanding under the first of four starting tokens, that's at least tens of thousands of somewhat-likely unique 10-token thinking rollouts. Generally, I associate this amount of diversity with *base models,* not chat models--for comparison, this is deepseek-v3 with the same partial thinking trace prefilled and same tree parameters: ...yeah. 2. R1 thinking traces are highly "reentrant." Despite this diversity, R1 returns to the same concepts over and over in different branches. It was actually extremely difficult to find a branch in this (massive) tree that *didn't* mention checking the documentation. Here are some examples of trajectories that all led to "checking the documentation": - Let me check the documentation - Let me check the PyQGIS documentation - Let me check. Looking at the QgsVertexMarker documentation - Let me check.\n\nLooking into QGIS documentation - Let me check. According to the QGIS documentation - Let me check.\n\nWait, looking at the documentation - Let me verify.\n\nLooking at the documentation - Wait, looking up the documentation - I need to check.\n\nLooking at the QGIS documentation You get the point. This has some interesting implications for pure token-based inference-time steering (think hfppl) of R1 thinking traces--I expect it would be very difficult to prevent R1 from taking a step it wants to take, and if you succeed, you may end up driving it into a very weird / marginal part of the distribution. 3. When R1 (rarely) didn't mention the [documentation, it was more vague... text continues on next screenshot] ——— 3. When R1 (rarely) didn't mention the documentation, it was more vague. When R1 "checked the documentation", it would only sometimes cite the exact constructor signature, and other times only state a fact about the constructor's behavior (e.g., that it adds the marker to the canvas). However (in the subtress I explored) when R1 *didn't* "check the documentation", it *never* cited the exact constructor, only more general facts. I have two theories about this: One is based on pretraining: this is a lot like how humans write in the corpus. When we check the docs, we tend to cite specifics, and when we're working from memory, we tend to only say what we can definitely remember that's directly relevant. If R1 is mimicking that behavior (which, after all, is most likely why it's pretending to check the docs in the first place), it would make sense why it's only specific when it's already said it's "checking the docs." My other theory is that this is an RL behavior: if R1 is less accurate about specifics when it hasn't "checked the docs", and inaccuracy in rollouts leads to wrong answers leads to low reward, perhaps it learns to steer away from specifics unless they're "licensed" by something that makes them more likely to be accurate, like [text continues on next screenshot] ```