Calc Consulting

Calc Consulting @CalcCon · Jul 2 ww-pgd [1 reply, 96 views] Fabian Franz @fabianfranz · Jul 2 @grok ELI5 on ww-pgd [1 reply, 38 views] Grok @grok · Jul 2 ELI5: ww-pgd = WeightWatcher Projected Gradient Descent. It's a PyTorch add-on that wraps normal optimizers (AdamW etc.). While the main trainer updates weights normally, ww-pgd occasionally "projects" them: it checks the spectral shape of each layer with WeightWatcher and gently nudges the heavy tail of singular values toward the sweet-spot power-law (α ≈ 2). Think of it as a spectral coach that stops AdamW from overfitting (α dropping too low) and Muon from underfitting (α staying too high). Keeps the model in the theoretical Goldilocks zone for better generalization and robustness.

Detecting signatures of overfitting in GPT2 using Random Matrix Theory (RMT) Here, we train GPT2, with both gradient clipping and weight decay, but even then after some period of time it overfits its training data, and the training accuracy traps. In other words, it anti-groks. Then we apply RMT to the layers weight matrices W. First, we randomize each W elementwise to remove the correlations. W-> rand(W) Then, we compute each layer eigenvalues of rand(W) and fit them to the Marchenko-Pastur (MP) distribution Finally, count the number eigenvalues that lie outside the MP fit. That is, the large outliers. We call these outliers Correlation Traps. Such traps screw up the correlated model, and keep it from generalizing well. and the larger the outlier, the bigger an effect they have. For the stat-phys folks, the MP fit is the self-averaging baseline, and the traps are non-self-averaging outliers. For the ML theory-minded, the traps have non-vanishing variance of order O(1), and cause the test error to lose concentration, therefore violating any generalization bound you can think of. And all of this can be done without needing access to the training or test data. All you need are the weights. You can see the signatures of harmful overfitting directly in the weight matrices. [image] --- Here's the broader story. We can see Correlation Traps in foundation-scale open-source models like OSS 20B and 120B. And there's a lot of them. [image] --- ##### Comments > **Gabe @Gabogonzalez515** · [2026-05-05](https://x.com/Gabogonzalez515/status/2051715179176710251) > > this is very similar to WeightWatcher hmmm > > > **Calc Consulting @CalcCon** · [2026-05-05](https://x.com/CalcCon/status/2051718726152802549) > > > > This is weightwatcher. > > > > I am weightwatcher > > > > [image] > **Alex Stern @alexkrstern** · [2026-05-05](https://x.com/alexkrstern/status/2051705574463525375) > > Would love to read this paper > > > **Calc Consulting @CalcCon** · [2026-05-05](https://x.com/CalcCon/status/2051711547819561119) > > > > Writing it up now > > > > I actually wrote a blog post on this several years ago. The technique has been in the tool for some time. > > > > Just getting around now to going through all the messy details > > > > You can join us on discord to learn more