Features
Native inference server for Apple Silicon (M1-M4)
OpenAI and Anthropic compatible API endpoints
Speculative decoding (PLD, cross-attention drafter, MTP)
Adaptive gates to disable speculation on novel content
DeepSeek V4 Flash (284B) support on 96GB+ Macs
Agent mode and MCP tool calling
Photo editing with natural language
Image-to-video and talking-character video
Voice cloning and Krea-2 image generation
Style LoRAs and DiffusionGemma
Model browser with one-click download of GGUF/MLX
Python API to define LLMs
C++ runtime for performant inference
Specialized kernels for common operations
Speculative decoding (including N-gram)
Sparse attention for long-context inference
Skip softmax attention for very long sequences
MoE communication optimization via one-sided AlltoAll over NVLink
Expert parallelism scaling
Guided decoding combining CPU and GPU
Visual generation (diffusion models) support
Day-0 support for new models (e.g., GPT-OSS, EXAONE)
Distributed weight data parallelism (DWDP) for NVL72
Tuning CUDA Graph batch sizes
Inference-time compute implementation
