Bitsandbytes

Bitsandbytes is the default quantization library for PyTorch users in the Hugging Face ecosystem. It is the easiest way to cut memory usage for LLM inference and training — often requiring just a single line change to load models in 4-bit or 8-bit. The 8-bit optimizers are a standout: they reduce optimizer memory by 75% while matching full-precision performance, which is huge for fine-tuning large models. For inference, LLM.int8() handles outliers gracefully, so you don't lose accuracy. QLoRA has become a standard for efficient fine-tuning. That said, bitsandbytes is not a one-size-fits-all solution. It only supports CUDA, so no AMD or Apple Silicon for training (though some progress has been made on ROCm). If you need post-training quantization (like GPTQ or AWQ) for ultra-low-bit models on consumer hardware, bitsandbytes doesn't offer that — it is focused on runtime quantization during inference or training. For pure inference speed, libraries like vLLM or TensorRT-LLM may outperform bitsandbytes for large batches or production serving. But for single-GPU or memory-constrained setups, bitsandbytes remains the most practical choice. We'd reach for it when fine-tuning Llama 3 or Mistral on a single 24GB GPU, or when deploying a chatbot on a consumer card. Pass if you need cross-platform or production-scale inference.