Features
PagedAttention for efficient memory management
Continuous batching for high throughput
OpenAI-compatible API for instant integration
Prefix caching for reduced latency
Speculative decoding for faster generation
Multi-GPU serving via tensor parallelism
Quantization support (NVFP4, BF16, MXFP8, AWQ, GPTQ)
Model runner V2 for iterative denoising (DiffusionGemma)
Semantic Router for micro-agent routing and fusion
vLLM-Omni for multi-stage multimodal serving (TTS, video)
Staged serving (Thinker-Talker-Code2Wav)
Support for sparse attention (MiniMax M3)
Integration with RL training via vime
Multi-hardware support (CUDA, ROCm, XPU, CPU, Apple Silicon, AWS Neuron, Google TPU, Huawei Ascend, IBM Spyre)
Distributed model training across GPU clusters
Multimodal data curation (video, images, text, audio)
Batch embedding generation with Sentence Transformers
Post-training LLM inference with vLLM and SGLang
Elastic scaling with last-mile data preprocessing
Fine-grained hardware allocation (CPU, GPU, TPU, NVL72)
Multi-cloud orchestration across GPU providers
Ray-native distributed object store and RDMA transport
Automatic cluster provisioning and scaling
GPU observability and advanced monitoring
Agent-first experience with Python APIs
Serverless execution with Python decorators
Bring Your Own Cloud (BYOC) deployment
Integration with PyTorch, vLLM, SGLang, XGBoost
On-premises deployment support via BYOC
