airllm

airllm is a fine-tuning toolkit - AirLLM 2.8T LLM inference with single 4GB GPU.

Overview

AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run Kimi K3 (2.8T) — the largest open-source model released to date — on under 4GB, Qwen3.8-Flash-Next (125B) on 6GB, and DeepSeek-V3 (671B) on ~12GB. We now also support training huge models on small VRAM: Qwen3.8-Flash-Next (125B) under 6GB.

[2026/08] Qwen3.8-Flash-Next support: Qwen's 125B MoE flagship ('Qwen4ExpForConditionalGeneration') with a ~51B n-gram embedding runs in 5.95GB of VRAM, measured end to end on one RTX 4090. The n-gram table is file-mapped on the host (a 64GB machine is enough); decoder layers stream. Needs a 'transformers' build with in-tree 'qwen4_exp' ('pip install git+https://github.com/huggingface/transformers.git' today) and ~360GB of checkpoint disk ('delete_original=True' reclaims the originals after the split).

[2026/08] Qwen3.8-27B support: Qwen's new dense VL (Gated DeltaNet + Gated Attention, native vision) runs in 3.33GB of VRAM, measured end to end on one RTX 3090. Needs 'transformers' 5.8+.

airllm is open-source, written primarily in Jupyter Notebook, with 38,700 GitHub stars under the Apache 2.0 license. The latest release is v4.0.0 (2026-09-05).

Key capabilities

From the project's documentation:

• Best AI Game Sprite Generator
• Best AI Facial Expression Editor
• Step 2. make sure airllm verion later than 2.0.0: pip install -U airllm
• Step 3. when initialize the model, passing the argument compression ('4bit' or '8bit'):
• profiling_mode: supported options: True to output time consumptions or by default False
• layer_shards_saving_path: optionally another path to save the splitted model

Install

A quick way to get started (always check the official docs for the latest):

pip install airllm

How it fits a local-AI stack

airllm runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related fine-tuning toolkits in the directory:

• Axolotl
• "ColossalAI"
• "DeepSpeed"
• "gitleaks"

Sources

• Source code & docs: lyogavin/airllm

Stats from GitHub, 2026-10-15.