paras chopra

Learned something very interesting today! Random projections of a non-linearly separable data onto high dimensional spaces is enough to make it linearly separable. Consider a dataset like XOR that you can't linearly separate. Now, if you project each 2D point onto a D (=50) dimensional space using *randomly* initialised basis vectors, each direction creates a tiny difference between the classes (e.g. gives 51-52% accuracy) because expectation of two classes differs slightly when randomly projected. So each randomly projected feature becomes a tiny discriminator and when you aggregate it over 20-50 such discriminators, a linear classifier is able to separate them perfectly by simply learning how much to weigh each feature. One intriguing possibility of this is that we're able to train deep networks because random projections make most of the data already separable, making the job of gradient descent easy. [Embedded image: four-panel figure. Panel 1 "XOR-ish data (not linearly separable)" shows a 2D scatter plot with four quadrants colored purple/yellow in checkerboard pattern. Panel 2 "Linear boundary on raw x, acc=0.514" shows the same data with a near-random linear split. Panel 3 "Random ReLU(D=50) + linear boundary, acc=0.989" shows the same data now separated almost perfectly by a complex boundary. Below, a line chart "XOR-ish: accuracy improves as D increases" showing test accuracy rising sharply from ~0.65 at D=0 to ~0.99 by D=50 and staying flat through D=200.]