vllm

Ahmad @TheAhmadOsman · 8h today i learned that cuda graphs on vllm v0 is 256 batch size and in v1 is 512, so that's why you get out of memory from cuda graphs compiling [Embedded conversation screenshot:] Replying to Ahmad the cuda graphs compiler size literally quadrupled for my runs in my cluster, using 8x 3090s for a big model (e.g. 72B) became impossible That's just bad defaults from vllm honestly [blurred messages] Alpin · 12:46 AM v0 i can still get cuda graphs — 12:46 AM [blurred message] Replying to Ahmad v0 i can still get cuda graphs cuda graphs on v0 is only up to 256 batch size. v1 extends that to 512 Alpin · 12:46 AM [Quoted older tweet:] Ah... @TheAhmadOs... · Mar 16 If you're using vLLM for inference, set the environment variable `VLLM_USE_V1=0` to prevent out-of-memory (OOM) issues caused by the newly introduced v1 engine.

gm8xx8 @gm8xx8 · 5m THE BLUEPRINT FOR ECOSYSTEM TRUST: - modular components - contributing optimizations upstream - syncing model releases w/ community from Day 0 open infra that scales. TAKE NOTES ↓ > QUOTED: Teortaxes▶️(Deep... @t... · 1h > DeepSeek Open Infra update: DeepSeek announces open-sourcing their inference engine, in collaboration with @lmsysorg SGLang and @vllm_project, by essentially porting it... Show more > [Screenshot of a GitHub README: "open-infra-index / OpenSourcing_DeepSeek_Inference_Engine / README.md", titled "The Path to Open-Sourcing the DeepSeek Inference Engine". Text: "A few weeks ago, during Open Source Week, we open-sourced several libraries. The response from the community has been incredibly positive - sparking inspiring collaborations, productive discussions, and valuable bug fixes. Encouraged by this, we've decided to take another step forward: contributing our internal inference engine back to the open-source community. We are deeply grateful for the open-source ecosystem, without which our progress toward AGI would not be possible. Our training framework relies on PyTorch, and our inference engine is built upon vLLM, both of which have been instrumental in accelerating the training and deployment of DeepSeek models. Given the growing demand for deploying models like DeepSeek-V3 and DeepSeek-R1, we want to give back to the community as much as we can. While we initially considered open-sourcing our full internal inference engine, we identified several challenges: Codebase Divergence: Our engine is based on an early fork of vLLM from over a year ago... Infrastructure Dependencies... Limited Maintenance Bandwidth... Considering these challenges, we've decided to collaborate with the SGLang project and vLLM project as more sustainable alternatives. SGLang and vLLM stand out for their broad hardware support, extensive model compatibility, and state-of-the-art performance... Moving forward, we will work closely with the SGLang team and vLLM team to: Extract Standalone Features... Share Optimizations... We are profoundly grateful for the open-source movement... Note: To clarify, this article outlines our approach to open-sourcing of our DeepSeek-vLLM codebase only. Regarding future model releases, we maintain an open and collaborative stance towards both the open-source community and hardware partners. We commit to proactively synchronizing inference-related engineering efforts prior to new model launches, with the goal of enabling the community to achieve state-of-the-art (SOTA) support from Day-0..."]