James Chen

James Chen @jchencxh · Mar 19 The actual implementation I give for doing this on a MIM task is actually very straightforward. You just take the hidden representations after every level of a ViT, sample some subset, and use each representation separately to predict masked patches at every single level. 1 reply, 9 likes, 454 views James Chen @jchencxh · 23h There are actually two reasons for predicting lower level abstractions: 1) less spuriousness (harder to be spurious for predicting a hierarchy) 2) retain lower level reps that can be useful for later composition in the net Note that for 2), it's because we're only communicating across representations levels through a single embedding through a feed forward net. If there was a larger communication bandwidth across the representation levels (e.g., some version of attention residuals), this problem could go away, and there's a lot more possibility for the higher level abstractions we can compose (as it's becomes easier to compose low and very high levels of abstraction, where the very lower level of abstraction would not be kept in a standard feed forward structure). I think that a newer class of architectures is probably better to exploit this type of learning over the hierarchy.