policy gradients

gm8xx8 @gm8xx8 · Apr 3 DGPO uses probability gradients instead of log-probability gradients in RLVR. Log-probability–based soft clipping becomes unstable as token probabilities vanish, with gradient weights blowing up at the left boundary. DGPO applies asymmetric decay to clipped tokens: polynomial on low-ratio for stability, reciprocal-radical on high-ratio to continue updating low-probability tokens instead of zeroing them out. This yields a smoother stability–learning tradeoff than clipping-based approaches. Across DeepSeek-R1-Distill-Qwen 1.5B, 7B, and 14B, it consistently improves mathematical reasoning benchmarks, positioning probability-gradient control as the cleaner RLVR primitive. [image: comparison diagrams and table of policy optimization algorithms (GRPO, CISPO, GPPO, CE-GPPO, ASPO, DGPO) showing gradient explosion/convergence plots and a table of properties: Exploration Capability, Gradient Convergence, Gradient Continuity, Exploration Convergence, Gradient Estimator Bias, Logging Token Update Magnitude across Left/Right/Reverse boundaries]