Features
Pipeline API for inference across 100+ tasks
Trainer with mixed precision, torch.compile, FlashAttention
Fast text generation with generate API (streaming, multiple decoding strategies)
Support for text, computer vision, audio, video, multimodal models
Integration with PyTorch, TensorFlow, and JAX
Parameter-efficient fine-tuning via PEFT integration
Quantization with bitsandbytes
Distributed training with DeepSpeed and FSDP
Model sharing and loading from Hugging Face Hub
Automatic mixed precision training
OpenAI-compatible API for model serving
Deploy 500+ open-source models
Write custom GPU kernels with Mojo
Zero dependency on PyTorch, CUDA, or ROCm
Single container under 700MB for self-hosted
Paged KV cache for memory efficiency
Quantization (bfloat16, float32)
Multi-node distributed inference
Model customization via PyTorch-like API
Hardware-agnostic (NVIDIA, AMD, Apple Silicon)
Mixture-of-Experts serving
max benchmark tool adapted from vLLM
MiniMax M3 open weights support
SOTA MoE serving on Modular Cloud
