Bitsandbytes

bitsandbytes is an open-source, MIT-licensed PyTorch library that shrinks the memory footprint of large language models through k-bit quantization — 8-bit and 4-bit weights and optimizer states. It targets researchers, hobbyists, and small teams who want to train or run LLMs on GPUs they already own, plus anyone working inside the Hugging Face stack who needs a drop-in memory reduction without rewriting model code. Current docs cover the mainline release v0.50.2 alongside v0.49.2, v0.48.2 and older branches. Three techniques do the work. 8-bit optimizers (AdaGrad, Adam, AdamW, AdEMAMix, LAMB, LARS, Lion, RMSprop, SGD) apply block-wise quantization to hold roughly 32-bit performance at a small fraction of the memory cost. LLM.int8() enables inference at about half the memory with no reported performance degradation, quantizing most features to 8-bit through vector-wise quantization and handling outliers separately in 16-bit matrix multiplication. QLoRA quantizes the base model to 4-bit and inserts a small set of trainable low-rank adaptation weights so training stays feasible on a single card, with FSDP-QLoRA extending that to distributed runs. It is PyTorch-only, and the docs state that full functionality requires NVIDIA GPUs. Because it is the default quantization backend across the Hugging Face ecosystem, you have probably already run it without naming it — loading a model in 8-bit or fine-tuning with QLoRA goes through bitsandbytes by default in Transformers and PEFT. The positioning is straightforward. Compared with post-training quantizers built for serving like GPTQ or AWQ, and against high-throughput inference servers like vLLM or TensorRT-LLM, bitsandbytes optimizes for getting a model to fit and train on constrained hardware rather than for maximum requests per second. It costs nothing and there is no vendor account to create.