Ahmad @TheAhmadOsman
— quoting a DM/thread with Alpin
Ahmad @TheAhmadOsman · 8h
today i learned that cuda graphs on vllm v0 is 256 batch size and in v1 is 512, so that's why you get out of memory from cuda graphs compiling
[Embedded conversation screenshot:]
Replying to Ahmad
the cuda graphs compiler size literally quadrupled for my runs in my cluster, using 8x 3090s for a big model (e.g. 72B) became impossible
That's just bad defaults from vllm honestly
[blurred messages]
Alpin · 12:46 AM
v0 i can still get cuda graphs — 12:46 AM
[blurred message]
Replying to Ahmad
v0 i can still get cuda graphs
cuda graphs on v0 is only up to 256 batch size. v1 extends that to 512
Alpin · 12:46 AM
[Quoted older tweet:]
Ah... @TheAhmadOs... · Mar 16
If you're using vLLM for inference, set the environment variable `VLLM_USE_V1=0` to prevent out-of-memory (OOM) issues caused by the newly introduced v1 engine.
Note from Claude Sonnet 5
A technical troubleshooting thread about vLLM inference engine CUDA graph batch-size defaults (v0: 256, v1: 512) causing out-of-memory errors when compiling CUDA graphs for large models on multi-GPU clusters. Practical ML infrastructure/ops content — relevant if Nathan uses vLLM for inference workloads on the training server.
machine learningvllmcuda graphsinference engineeringgpu memorytwittertechnical