Features
On-device LLM inference with X-Bit quantization (sub-4-bit)
No cloud dependency; all data stays on device
Real-time inference for voice and text applications
Integrates with Picovoice voice AI stack (wake word, STT, TTS)
Supports RAG (Retrieval-Augmented Generation) for document QA
Low latency and offline operation
Privacy-preserving: no data leaves the device
SDK for multiple platforms (Android, iOS, Linux, macOS, Windows, Web, Python)
Customizable model compression with picoCompression
Benchmarked against GPTQ, GGUF, and SpinQuant for accuracy/speed
OpenAI-compatible API for model serving
Deploy 500+ open-source models
Write custom GPU kernels with Mojo
Zero dependency on PyTorch, CUDA, or ROCm
Single container under 700MB for self-hosted
Paged KV cache for memory efficiency
Quantization (bfloat16, float32)
Multi-node distributed inference
Model customization via PyTorch-like API
Hardware-agnostic (NVIDIA, AMD, Apple Silicon)
Mixture-of-Experts serving
max benchmark tool adapted from vLLM
MiniMax M3 open weights support
SOTA MoE serving on Modular Cloud
