Features
Local inference on Apple Silicon (M-series chips)
Model conversion and compression
Quantization schemes co-designed with architecture
Hardware-aware tensor operation optimization
Batch size = 1 inference engine
Real-time decoding for interactive AI
Apple device automation integration
Models library for pre-optimized models
Sparse buffers for KV cache (June 2026)
New quantization method for speed-quality frontier (June 2026)
Fast & lossless inference for 1-bit LLMs (BitNet b1.58)
Optimized CPU kernels for ARM & x86 architectures
Official GPU inference kernel (released 05/2025)
Parallel kernel implementations with configurable tiling
Embedding quantization for 1.15x–2.1x additional speedup
1.37x–6.17x CPU speedup vs baseline
55%–82% CPU energy reduction
Run 100B BitNet b1.58 on single CPU (5-7 tok/s)
Lookup Table kernels built on T-MAC methodologies
Support for Hugging Face 1-bit models
Conda environment setup script (setup_env.py)
Inference server (run_inference_server.py)
Lossless inference—no accuracy degradation
Falcon3 family and Llama3-8B-1.58 model support
Supports models up to 100B parameters
