Features
PagedAttention for efficient memory management
Continuous batching for high throughput
OpenAI-compatible API for instant integration
Prefix caching for reduced latency
Speculative decoding for faster generation
Multi-GPU serving via tensor parallelism
Quantization support (NVFP4, BF16, MXFP8, AWQ, GPTQ)
Model runner V2 for iterative denoising (DiffusionGemma)
Semantic Router for micro-agent routing and fusion
vLLM-Omni for multi-stage multimodal serving (TTS, video)
Staged serving (Thinker-Talker-Code2Wav)
Support for sparse attention (MiniMax M3)
Integration with RL training via vime
Multi-hardware support (CUDA, ROCm, XPU, CPU, Apple Silicon, AWS Neuron, Google TPU, Huawei Ascend, IBM Spyre)
OpenAI-compatible API for model serving
Deploy 500+ open-source models
Write custom GPU kernels with Mojo
Zero dependency on PyTorch, CUDA, or ROCm
Single container under 700MB for self-hosted
Paged KV cache for memory efficiency
Quantization (bfloat16, float32)
Multi-node distributed inference
Model customization via PyTorch-like API
Hardware-agnostic (NVIDIA, AMD, Apple Silicon)
Mixture-of-Experts serving
max benchmark tool adapted from vLLM
MiniMax M3 open weights support
SOTA MoE serving on Modular Cloud
