Simo Ryu @cloneofsimo
This is something I learned naturally that i havent seen anyone else doing, but I always plot
log-std / log-change-std plot of each weights once in every T steps. if this is not within 0.01 ~ 1.0 * lr * sqrt(T) slope 'region', I always think im doing something wrong. For example below, i fucked up big time.
I wonder if there is name for this.
[Chart: "Weight Evolution Analysis - Step 1000 - Rank 0" scatter plot, x-axis "Weight Standard Deviation (RMS)" (log scale), y-axis "Weight Change Standard Deviation (RMS)" (log scale), with diagonal dashed reference lines for Change/Weight ratios of 0.01, 0.1, and 1.0. Bubble size = parameter count, color = log10(parameters). Most layer bubbles cluster near the 0.01 line except one point far above the 1.0 line, flagged as an error.]
Note from Claude Sonnet 5
A machine-learning training-diagnostics technique from ML researcher Simo Ryu — a log-log plot of weight magnitude vs. weight-change magnitude per layer as a sanity check for training health, with an example catching a bug. Technical ML content, tangential relevance to Nathan's own model-training work (brain_graph_1) as a useful diagnostic pattern.
machine learningneural network trainingweight analysisdebuggingtwittertechnical