tandem.cpp
llama.cpp patch: faster MoE offload for models that don't fit in VRAM
What it does
Core capabilities at a glance
- Gguf
- Inference Engine
- Llama CPP
- Local LLM
- MOE
Deep dive
The full breakdown - performance, comparisons, and setup
tandem.cppLink to this section
tandem.cpp is a local inference server - llama.cpp patch: faster MoE offload for models that don't fit in VRAM.
OverviewLink to this section
tandem.cpp is a patch set for llama.cpp. It makes MoE models that don't fit in VRAM run faster in llama.cpp. It keeps the most-used experts on the GPU and lets the CPU help with the rest, for any MoE model llama.cpp supports. Single-user decode is the target.
Decode tok/s, one user, GSM8K questions. Each row is one model on one PC. Two numbers are greedy / the model's recommended sampling, one number is greedy.
The 32 GB row is a cold start from a DRAM-less PCIe 4.0 SSD; at 64 GB DeepSeek-V4.1 is about twice the RAM and is read from a PCIe 4.0 SSD.
We tuned for Qwen3.8 only, but other MoE models that do not fit in VRAM gain too: typically 1.4–2.1× (gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1), less when the model nearly fits. Prompts are also 4–13× faster than llama.cpp's on a 16 GB card with the experts in RAM or on the SSD (v0.3.0, v0.4.0).
Strata is a dedicated MoE offload engine for a few models. Its default skips low-ranked experts ('STRATA_ROUTE_TAIL_SKIP'), which is faster but approximate (127.1 tok/s in the RTX 4090 Qwen3.8 row); the tables show its exact mode ('STRATA_ROUTE_TAIL_SKIP=0', every routed expert computed, as llama.cpp and tandem.cpp do). Full tables per machine are below.
tandem.cpp is open-source, written primarily in Python, with 108 GitHub stars under the Other license. It was last updated on 2026-11-20.
Key capabilitiesLink to this section
From the project's documentation:
- Long prompts faster again: layer streaming. A prompt batch of more than 1024 tokens uses every expert of every layer,
- Warm start (documented now, works with the faster prompts): LLAMA_MOE_HOT_SAVE=hot.txt writes the experts a session
- Prompts 2–6× faster. With an expert cache, prompts now run in 4096-token batches instead of 512. Each batch has to
- Store mode: when the RAM store is a few slots short of holding every expert next to the VRAM cache, it now holds all but
- A big card in a small PC: VRAM and RAM hold every expert. When the VRAM cache plus the RAM store can hold all the
- Faster first token in store mode. Each expert upload's padding was read from the file, an SSD read per expert.
How it fits a local-AI stackLink to this section
tandem.cpp runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:
SourcesLink to this section
- Source code & docs: ewhacc/tandem.cpp
Stats from GitHub, 2026-11-21.
Frequently asked
Quick answers to common questions
What is tandem.cpp?
tandem.cpp is a inference-server tool for local AI workloads. llama.cpp patch: faster MoE offload for models that don't fit in VRAM
Is tandem.cpp free and open source?
Yes, tandem.cpp has 108 GitHub stars and is licensed under Other. You can self-host it for free on .
What hardware do I need for tandem.cpp?
The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. tandem.cpp has 108 GitHub stars and an active community.
Does tandem.cpp support GPU acceleration?
tandem.cpp's GPU support depends on your specific setup. Check the documentation for details. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.
What are the best alternatives to tandem.cpp?
Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.
How much does tandem.cpp cost?
tandem.cpp is free-open-source. It is completely free and open source to self-host.
Pairs well with
Complementary tools, models, and hardware