deep learning paradigms

Zeeshan Patel @zeeshanp_ · 1h after leaving frontier labs, many technical staff choose to build new companies. you'd imagine that it would be hard without large-scale data and compute to work on frontier research. if anything, we're seeing the complete opposite play out. there are several "neolabs" working on novel research and making good progress at small scale. even neolabs that raise hundreds of millions usually only have a few thousand chips at most, which is trivial compared to the incumbents. the key insight is that you don't need large data or compute to make meaningful progress. it's easy to forget that the core breakthroughs powering the industry today were discovered with extremely scarce resources by modern standards. to make fundamental developments, it's important to spend time finding more effective ways of utilizing compute rather than just scaling existing paradigms. many times, this is best executed under tighter resource constraints. it's very exciting to see so many talented folks taking courageous next steps towards researching new frontiers, which will hopefully bring upon new paradigms in deep learning.