Features
PagedAttention for efficient memory management
Continuous batching for high throughput
OpenAI-compatible API for instant integration
Prefix caching for reduced latency
Speculative decoding for faster generation
Multi-GPU serving via tensor parallelism
Quantization support (NVFP4, BF16, MXFP8, AWQ, GPTQ)
Model runner V2 for iterative denoising (DiffusionGemma)
Semantic Router for micro-agent routing and fusion
vLLM-Omni for multi-stage multimodal serving (TTS, video)
Staged serving (Thinker-Talker-Code2Wav)
Support for sparse attention (MiniMax M3)
Integration with RL training via vime
Multi-hardware support (CUDA, ROCm, XPU, CPU, Apple Silicon, AWS Neuron, Google TPU, Huawei Ascend, IBM Spyre)
Fast & lossless inference for 1-bit LLMs (BitNet b1.58)
Optimized CPU kernels for ARM & x86 architectures
Official GPU inference kernel (released 05/2025)
Parallel kernel implementations with configurable tiling
Embedding quantization for 1.15x–2.1x additional speedup
1.37x–6.17x CPU speedup vs baseline
55%–82% CPU energy reduction
Run 100B BitNet b1.58 on single CPU (5-7 tok/s)
Lookup Table kernels built on T-MAC methodologies
Support for Hugging Face 1-bit models
Conda environment setup script (setup_env.py)
Inference server (run_inference_server.py)
Lossless inference—no accuracy degradation
Falcon3 family and Llama3-8B-1.58 model support
Supports models up to 100B parameters
