@mkurman88

##### Conversation[Mariusz Kurman](https://x.com/mkurman88)[@mkurman88](https://x.com/mkurman88) LR 1e-4, and training loss stays as is, ranging from 1.5 to 1.55. Eval loss acts in the same way. What do you do? Here is what I did (NeuroBLAST v3 to v4 migration): First I did not assume the LR was the problem. A flat loss at 1.5–1.55 can mean many things: bad LR, bad data, architecture ceiling, or simply some part of the model not actually updating. I looked up the data previously and did some ablations with different LRs without success, so I treated it as a learning-pathology/debugging problem. I compared checkpoints across training. Globally the model was changing, so it was not a fully dead run. But one very suspicious thing showed up: the two new NeuroBLAST v4 stage-transition LayerScale tensors that were meant to improve gradient flow, sensory\_to\_associative.scale and sensory\_to\_motor.scale, were byte-identical across checkpoints. They were stuck at around 0.1001. Then I checked whether they were disconnected. After reconstructing the exact order of optimizer parameters, I found that their Adam moments were nonzero. So this was not “no gradient”. The issue was numerical: the parameters and Adam moments were bf16. Around value 0.1, bf16 has spacing of about 4.88e-4, while the estimated Adam updates were only around 1e-5 to 6e-5. So every update rounded away. The tensors were receiving signal but could not move. The fix was surgical: keep those transition scale parameters in fp32, prevent “.to(bfloat16)” from recasting them, and promote their resumed optimizer state to fp32 as well. After resuming, the next checkpoint proved the diagnosis. At the next checkpoint, both stage-transition scales were saved as fp32, all 512 channels had changed, and the tensors were no longer uniform. sensory\_to\_associative.scale moved to roughly mean 0.10035, std 0.0056; sensory\_to\_motor.scale moved to mean 0.09809, std 0.0081. The optimizer moments were fp32 too. Finally I swept all tensors from checkpoint 31500 to 32000. No large matrices were frozen: attention projections, MLPs, conv kernels, embeddings, LM head, and stage scales were all moving. The only exact-zero tensors left were tiny norm/gain vectors, about 0.015% of parameters, likely another bf16-resolution artifact rather than a structural learning failure. So when loss plateaued, I didn’t just lower LR or restart. I audited whether the architecture was really learning. The plateau exposed a precision bug in the new stage-transition mechanism. After fixing fp32 storage for those tiny but important scale vectors, the previously frozen architectural path started learning.[Mariusz Kurman](https://x.com/mkurman88)[@mkurman88](https://x.com/mkurman88) Symptom visible here Quote Mariusz Kurman @mkurman88 Vibe-training. Optimizer can't decide which minima to choose [[image]](https://x.com/mkurman88/status/2070806105085943976/photo/1)