bitsandbytes

bitsandbytes is a fine-tuning toolkit - Accessible large language models via k-bit quantization for PyTorch.

Overview

'bitsandbytes' enables accessible large language models via k-bit quantization for PyTorch. We provide three main features for dramatically reducing memory consumption for inference and training:

• 8-bit optimizers uses block-wise quantization to maintain 32-bit performance at a small fraction of the memory cost. * LLM.int8() or 8-bit quantization enables large language model inference with only half the required memory and without any performance degradation. This method is based on vector-wise quantization to quantize most features to 8-bits and separately treating outliers with 16-bit matrix multiplication. * QLoRA or 4-bit quantization enables large language model training with several memory-saving techniques that don't compromise performance. This method quantizes a model to 4-bits and inserts a small set of trainable low-rank adaptation (LoRA) weights to allow training.

The library includes quantization primitives for 8-bit & 4-bit operations, through 'bitsandbytes.nn.Linear8bitLt' and 'bitsandbytes.nn.Linear4bit' and 8-bit optimizers through 'bitsandbytes.optim' module.

• Python 3.10+ * PyTorch 2.4+ * Note: While we aim to provide wide backwards compatibility, we recommend using the latest version of PyTorch for the best experience.

Note: this table reflects the status of the current development branch. For the latest stable release, see the document in the 0.50.0 tag.

Platform Accelerator Hardware Requirements LLM.int8() QLoRA 4-bit 8-bit Optimizers

• While supported, these marked features may lack in performance optimizations.

bitsandbytes is open-source, written primarily in Python, with 8,528 GitHub stars under the MIT license. The latest release is 0.50.2 (2026-08-27).

Key capabilities

From the project's documentation:

• 8-bit optimizers uses block-wise quantization to maintain 32-bit performance at a small fraction of the memory cost.

How it fits a local-AI stack

bitsandbytes runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related fine-tuning toolkits in the directory:

• "airllm"
• Axolotl
• "ColossalAI"
• "DeepSpeed"

Sources

• Source code & docs: bitsandbytes-foundation/bitsandbytes
• Official website: https://huggingface.co/docs/bitsandbytes/main/en/index

Stats from GitHub, 2026-10-24.