exllama social preview
other2,944MIT

exllama

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

Updated Oct 31, 2026
GitHub Docs
Platforms
linux, windows, docker, web
Pricing
free-open-source
Status
active
License
MIT

Deep dive

The full breakdown - performance, comparisons, and setup

exllama

exllama is a local-AI tool - A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

Overview

A standalone Python/C++/CUDA implementation of Llama for use with 4-bit GPTQ weights, designed to be fast and memory-efficient on modern GPUs.

Disclaimer: The project is coming along, but it's still a work in progress!

I am developing on an RTX 4090 and an RTX 3090-Ti. 30-series and later NVIDIA GPUs should be well supported, but anything Pascal or older with poor FP16 support isn't going to perform well. AutoGPTQ or GPTQ-for-LLaMa are better options at the moment for older GPUs. ROCm is also theoretically supported (via HIP) though I currently have no AMD devices to test or optimize on.

pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu118

  1. Install MSVC 2022. You can choose to install the whole 'Visual Studio 2022' IDE, or alternatively just the 'Build Tools for Visual Studio 2022' package (make sure 'Desktop development with C++' is ticked in the installer), it doesn't really matter which. 2. Install the appropriate version of PyTorch, choosing one of the CUDA versions. I am developing on the nightly build, but the stable version (2.0.1) should also work. 3. Install CUDA Toolkit, (11.7 and 11.8 both seem to work, just make sure to match PyTorch's Compute Platform version). 4. For best performance, enable Hardware Accelerated GPU Scheduling.

exllama is open-source, written primarily in Python, with 2,944 GitHub stars under the MIT license. It was last updated on 2023-09-30.

Key capabilities

From the project's documentation:

  • Python 3.9 or newer
  • torch tested on 2.0.1 and 2.1.0 (nightly) with cu118

Install

A quick way to get started (always check the official docs for the latest):

docker run --gpus all -p 5000:5000 -v <path_to_model_dir>:/data/model/ -v <path_to_session_dir>:/data/exllama_sessions --rm -it exllama-web --host 0.0.0.0:5000

How it fits a local-AI stack

exllama runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local-AI tools in the directory:

Sources

Stats from GitHub, 2026-10-26.

Frequently asked

Quick answers to common questions

What is exllama?

exllama is a other tool for local AI workloads. A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

Is exllama free and open source?

Yes, exllama has 2,944 GitHub stars and is licensed under MIT. You can self-host it for free on linux, windows, docker, web.

What platforms does exllama support?

exllama runs on linux, windows, docker, web.

What hardware do I need for exllama?

The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. exllama has 2,944 GitHub stars and an active community.

Does exllama support GPU acceleration?

exllama supports GPU acceleration via CUDA, Metal, or Vulkan depending on your platform. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.

What are the best alternatives to exllama?

Popular alternatives include other other tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.

How much does exllama cost?

exllama is free-open-source. It is completely free and open source to self-host.

Pairs well with

Complementary tools, models, and hardware