neolab

Deedy (verified, small org badge) @deedydas · 1h
The economics of a Neolab.

A neolab is loosely defined as a startup of AI researchers who raises a lot of
money pre-production to be able to finance GPU compute to take on a
large AI problem.

To buy 1000 GB300s or ~14 NVL72 racks will set you back $125-150M for
3yrs with 15-30% upfront. That's about ~2-2.5MW.

Thats about enough to do 10^25 flops a quarter and get to a GPT-4 level
model which is 1-2 OOMs off frontier for pretraining.

If you post-train on a great open source model, you have a better chance of
getting to frontier. The risks are a) you need to spend millions on RL
environments too and b) being lapped by another model release while
being tied to a base model.

For this to payback, you need to give your customers a better and ideally
cheaper inference service than a base model and serve them for long
enough to recoup your large investment. Even at 50% margin on inference,
to recoup $10M in training means serving ~10T tokens (!) if you price like
Fable / Astra given a standard cache read / input / output split ($2/M
blended). And you have to justify being better than a release like Opus 5.5
which is even cheaper. Often, you end up charging your customers a huge
premium in terms of platform fees and compute fees on top of pure
inference.

Meanwhile, every hour you're not utilizing your GPUs you are burning money
so you typically resell this compute back to a broker or run inference for
open models / resell spot instances. At below a ~60% utilization on spot,
you will still lose money.

Add to that insane cost of talent.

So what can you do with the compute?
- Not play the model game at all.
- Play an entirely different model game (Jev, World Labs) that if big labs
played, would either a) cannibalize their business or b) be incrementally not
significant revenue c) would cause too much distraction from the main
main thing
- Acquire a proprietary data set (Periodic Labs) in enough volume in a
domain of usefulness to eclipse frontier quality. Often happens in robotics,
biology, chemistry.

If you do overcome the challenge of building a model that is useful and well
priced beyond big labs models, given the huge price of compute, you still
need to play in an area where the revenue / compute ratio is signficant and
market demand is large enough to payback your compute spend.

It is a difficult game.

neolabai economicsgpu computeinference pricingpost-trainingstartups