Features
High-performance LLM and multimodal inference
Disaggregated prefill/decode pipeline
Speculative decoding for faster generation
Optimized GPU kernels (FlashAttention, etc.)
OpenAI-compatible API interface
Single-command server launch
Install via pip or Docker
Multi-node and multi-GPU inference
Structured output and sampling parameters
Vision language model support (v0.4.0+)
Runs on NVIDIA, AMD, CPU, TPU, Ascend, XPU
Model support: DeepSeek, Qwen, Llama, Mistral, GLM, GPT-OSS
Community support on GitHub, Slack, Discord
Scalable from single GPU to distributed clusters
Command generative model: agentic, multimodal, 49 languages
North Mini Code open-source agentic coding model
Transcribe speech-to-text in 14 languages
Embed multimodal text representations for RAG
Rerank relevance boosting for search quality
Model Vault dedicated secure inference per instance
Private deployment in VPC, on-premises, or Model Vault
North enterprise AI workplace platform with agents
Compass intelligent search and discovery system
LLM serving fairness to prevent noisy neighbor effects
Customization via fine-tuning on proprietary data
Multi-layered security and industry certifications
API access with trial and production keys
Agentic tool orchestration and reasoning
Integration with Wiz for security incident automation via MCP
