Features
MetalRT: custom Metal GPU kernels for Apple Silicon
QHexRT: full-stack NPU inference for Qualcomm Hexagon
Open-source SDKs: Swift, Kotlin, React Native, Flutter, Web
On-device inference by default with automatic cloud routing
Mirar: real-time vision agent with local pre-filtering
LLM, VLM, STT, TTS, and speech-to-speech inference support
Fleet operations and OTA model updates via console
Sub-10ms local inference latency
Cost-aware policy engine for routing
Benchmark-backed performance with methodology disclosure
Integration with OpenRouter (300+ models)
Integration with vLLM, Gemini, GPT-4o, Claude
Custom endpoint integration support
Trace visibility for agent steps (prompts, retrievals, tool calls, outputs)
LLM-as-judge evaluation for relevance, toxicity, quality scoring
Dataset creation from traces for reproducible testing
Experiment management and regression benchmarking
Built-in Prompt IDE for iterative prompt optimization
Self-hosted deployment on local, Docker, Kubernetes
Phoenix Cloud managed hosting option
Vendor-agnostic support for any model/framework
Native OpenTelemetry integration
OpenInference specification for LLM telemetry
Human annotation and automated labeling
Ghost trajectories to simulate alternative agent paths
Eval-as-you-test for early quality feedback
One-click integration with LlamaIndex
