Features
High-performance LLM and multimodal inference
Disaggregated prefill/decode pipeline
Speculative decoding for faster generation
Optimized GPU kernels (FlashAttention, etc.)
OpenAI-compatible API interface
Single-command server launch
Install via pip or Docker
Multi-node and multi-GPU inference
Structured output and sampling parameters
Vision language model support (v0.4.0+)
Runs on NVIDIA, AMD, CPU, TPU, Ascend, XPU
Model support: DeepSeek, Qwen, Llama, Mistral, GLM, GPT-OSS
Community support on GitHub, Slack, Discord
Scalable from single GPU to distributed clusters
One-command install on macOS, Linux, Windows
Run hundreds of open models locally
MLX engine for Apple Silicon (faster, less memory)
GGUF model support via llama.cpp (Ollama 0.30)
NVIDIA Nemotron 3 Ultra for high-throughput reasoning
Cloud scaling with Free, Pro, Max tiers
Run multiple cloud models in parallel (1, 3, 10)
Web-enabled cloud agents for real-time info retrieval
Fully offline operation for mission-critical work
Data never used for training; privacy-first design
CLI tool with model management and configuration
REST API for building AI applications
Upload and share private models (Pro and above)
40,000+ community integrations
Desktop app for macOS, Linux, Windows
