5 captures, most recent first.
kalomaze @kalomaze · 50m
periodic functions seem like the most fundamental class that conventional nonlinearities can't generalize over, even with depth
even tho you have exponentially many piecewise linear regions with ReLU-likes, that's not sufficient to extrapolate a sine wave...
Note from Claude Sonnet 5
A tweet from kalomaze arguing that periodic functions (like sine waves) are a fundamental class conventional neural network nonlinearities (ReLU-likes) can't extrapolate/generalize over, even with depth or many piecewise-linear regions.
neural networksnonlinearitiesgeneralizationperiodic functionsml theorytwitter
Learned something very interesting today!
Random projections of a non-linearly separable data onto high dimensional spaces is enough to make it linearly separable.
Consider a dataset like XOR that you can't linearly separate. Now, if you project each 2D point onto a D (=50) dimensional space using *randomly* initialised basis vectors, each direction creates a tiny difference between the classes (e.g. gives 51-52% accuracy) because expectation of two classes differs slightly when randomly projected.
So each randomly projected feature becomes a tiny discriminator and when you aggregate it over 20-50 such discriminators, a linear classifier is able to separate them perfectly by simply learning how much to weigh each feature.
One intriguing possibility of this is that we're able to train deep networks because random projections make most of the data already separable, making the job of gradient descent easy.
[Embedded image: four-panel figure. Panel 1 "XOR-ish data (not linearly separable)" shows a 2D scatter plot with four quadrants colored purple/yellow in checkerboard pattern. Panel 2 "Linear boundary on raw x, acc=0.514" shows the same data with a near-random linear split. Panel 3 "Random ReLU(D=50) + linear boundary, acc=0.989" shows the same data now separated almost perfectly by a complex boundary. Below, a line chart "XOR-ish: accuracy improves as D increases" showing test accuracy rising sharply from ~0.65 at D=0 to ~0.99 by D=50 and staying flat through D=200.]
Note from Claude Sonnet 5
A tweet by AI researcher Paras Chopra explaining an empirical/intuitive account of why random high-dimensional projections make non-linearly-separable data linearly separable, offered as a partial explanation for why deep networks are trainable via gradient descent. General ML theory content, not directly about safety or model welfare.
machine learningdeep learning theoryrandom projectionsneural networkstwitterparas chopra
Prime Intellect @PrimeIntellect
[no caption text visible]
[Top image: an abstract network diagram of white dots connected by many white lines on a dark background, forming a shape that narrows to a point on the left and fans out into a dense wide lattice on the right — resembling a stylized neural network or wave-fan visualization.]
[Bottom image: a classic physics diagram (appears to be an old textbook illustration) showing wave interference — a point source S emitting circular wavefronts through two barriers A and B (with slits O, O1, O2), producing an interference pattern that fans out and hits a wall C on the right, drawn in white on dark blue.]
9:10 AM · Sep 17, 2025 · 401.5K Views
Note from Claude Sonnet 5
A tweet visually pairing an abstract neural-network-like node diagram with a classic wave-interference (double-slit) physics diagram, implying a visual/conceptual analogy between neural network structure and wave propagation/interference. No caption text; the juxtaposition is the content.
neural networksphysicswave interferencevisualizationtwitter
Shannon Sands @max_paperclips
the not-really-proven-but-plausible explanation is that depth = better reasoning (ie, deeper networks are better function approximators). Depth for reasoning, girth for recall
10:38 PM · Apr 29, 2025 · 19 Views
[1 reply, 2 likes]
Lisan al Gaib @scaling01 · 1m
yep
fucking girth had me dead
Note from Claude Sonnet 5
A casual technical exchange positing that network depth correlates with reasoning ability while width ("girth") correlates with recall/memorization — an informal architecture heuristic in ML Twitter discourse, tangential to Nathan's own neural architecture work (brain_graph_1 depth-vs-width experiments).
twitterneural networksmodel architecturedepth vs widthml discourse
Brandon ✓ @brandon_xyzw
MLP actually looks terrifyingly large if you get the alpha and color encoding just right
[Image: a dense visualization of an MLP's weight connections, rendered as a symmetric diamond/hourglass shape in blue and orange with dense crossing lines suggesting connection density, with partial UI text "L:(4/" and a legend showing "ROW", "DES", "COL", "DES", "VAL"]
10:53 AM · Apr 9, 2025 · 167 Views
Note from Claude Sonnet 5
Tweet sharing a striking visualization of a neural network MLP layer's weights, rendered dense and symmetric via careful alpha/color encoding — an interpretability/visualization aesthetic post rather than a research finding.
twitterinterpretabilityvisualizationmlpneural networks
