neural architecture

| Component | Trainable params | Source |al) Counted dire— ───────────────────────────────────────── s | params | | Sharded macrocolumn cortex (425,393 | 184,354,125 | banner param_count() over | | | macrocolumns) | | requires_grad | Counted dire— Component | Trainable ─────────────────────────────────────────s • ├ — — | 184,354,125 | ba ner param_count() over────────────| | Sharded macrocolumn cortex (425,393 | | requires_gradount() over | | | Backbone (RWKV nodes, MONET keys, heads, | 37,302,731 | optimizer exp_avg entries (one | | | BG/motor/hippo/grid/etc.) | | moment per optimized param) | | Total trainable | ≈ 221.7 M | The macrocolumn cortex is 88% of the trainable weights — as expected, it's the whole point of the sharded run (it's also why N=425k doesn't fit one 24 GB card and needs the 2-way split). N te on the bigger number you'd see h̶u̶ external-drive/ 5 shells • buffers: EMA target encoders, CSR/edge-tructure tensors, divnorm running-σ, fast-plastic • local_learning traces (13.6 M), surprise memory (4.8 M), etc. Those don't get gradients, so they're not in the "trainable" count. • - The macrocolumn block's own per-shard optim[izers] are built at runtime and aren't in thiso checkpoint's exp_avg, which is why I took its trainable count from the startup banner's param_count() stead — the two method cover disjoint halves, so they sum cleanly. So: ~222 M trainable, ~184 M of it in the GPU-sharded cortex, with another ~73 M of non-trained [buffers/state cut off at bottom of visible screen]

James Chen @jchencxh · Mar 19 The actual implementation I give for doing this on a MIM task is actually very straightforward. You just take the hidden representations after every level of a ViT, sample some subset, and use each representation separately to predict masked patches at every single level. 1 reply, 9 likes, 454 views James Chen @jchencxh · 23h There are actually two reasons for predicting lower level abstractions: 1) less spuriousness (harder to be spurious for predicting a hierarchy) 2) retain lower level reps that can be useful for later composition in the net Note that for 2), it's because we're only communicating across representations levels through a single embedding through a feed forward net. If there was a larger communication bandwidth across the representation levels (e.g., some version of attention residuals), this problem could go away, and there's a lot more possibility for the higher level abstractions we can compose (as it's becomes easier to compose low and very high levels of abstraction, where the very lower level of abstraction would not be kept in a standard feed forward structure). I think that a newer class of architectures is probably better to exploit this type of learning over the hierarchy.