Features
Adaptive Inference – auto-fine-tunes from production failures
Automatic failure clustering per task and route
One-line integration with OpenAI/Claude SDKs
50+ models including Qwen, DeepSeek, Gemma, Nemotron, GLiNER
Model Router – intelligently routes tasks to best model
Continuous LoRA retraining from live traffic
Full PDF report per auto-agent run
Download model weights and training datasets
Streaming, tool calls, and structured outputs
Fine-tuning agent – describe task in plain English
Built-in evals and regression testing
Real-time latency and accuracy monitoring dashboard
GLiNER2-PII: open-source privacy filtering with PII detection
GLiGuard: 16x faster safety moderation with small language model
Fast & lossless inference for 1-bit LLMs (BitNet b1.58)
Optimized CPU kernels for ARM & x86 architectures
Official GPU inference kernel (released 05/2025)
Parallel kernel implementations with configurable tiling
Embedding quantization for 1.15x–2.1x additional speedup
1.37x–6.17x CPU speedup vs baseline
55%–82% CPU energy reduction
Run 100B BitNet b1.58 on single CPU (5-7 tok/s)
Lookup Table kernels built on T-MAC methodologies
Support for Hugging Face 1-bit models
Conda environment setup script (setup_env.py)
Inference server (run_inference_server.py)
Lossless inference—no accuracy degradation
Falcon3 family and Llama3-8B-1.58 model support
Supports models up to 100B parameters
