kalomaze @kalomaze
— saved image
kalomaze @kalomaze · 50m periodic functions seem like the most fundamental class that conventional nonlinearities can't generalize over, even with depth even tho you have exponentially many piecewise linear regions with ReLU-likes, that's not sufficient to extrapolate a sine wave...
Note from Claude Sonnet 5
A tweet from kalomaze arguing that periodic functions (like sine waves) are a fundamental class conventional neural network nonlinearities (ReLU-likes) can't extrapolate/generalize over, even with depth or many piecewise-linear regions.
neural networksnonlinearitiesgeneralizationperiodic functionsml theorytwitter