tandem.cpp social preview
inference-server108Other

tandem.cpp

llama.cpp patch: faster MoE offload for models that don't fit in VRAM

Updated Nov 21, 2026
Platforms
Pricing
free-open-source
Status
active
License
Other

What it does

Core capabilities at a glance

  • Gguf
  • Inference Engine
  • Llama CPP
  • Local LLM
  • MOE

Deep dive

The full breakdown - performance, comparisons, and setup

tandem.cppLink to this section

tandem.cpp is a local inference server - llama.cpp patch: faster MoE offload for models that don't fit in VRAM.

OverviewLink to this section

tandem.cpp is a patch set for llama.cpp. It makes MoE models that don't fit in VRAM run faster in llama.cpp. It keeps the most-used experts on the GPU and lets the CPU help with the rest, for any MoE model llama.cpp supports. Single-user decode is the target.

Decode tok/s, one user, GSM8K questions. Each row is one model on one PC. Two numbers are greedy / the model's recommended sampling, one number is greedy.

The 32 GB row is a cold start from a DRAM-less PCIe 4.0 SSD; at 64 GB DeepSeek-V4.1 is about twice the RAM and is read from a PCIe 4.0 SSD.

We tuned for Qwen3.8 only, but other MoE models that do not fit in VRAM gain too: typically 1.4–2.1× (gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1), less when the model nearly fits. Prompts are also 4–13× faster than llama.cpp's on a 16 GB card with the experts in RAM or on the SSD (v0.3.0, v0.4.0).

Strata is a dedicated MoE offload engine for a few models. Its default skips low-ranked experts ('STRATA_ROUTE_TAIL_SKIP'), which is faster but approximate (127.1 tok/s in the RTX 4090 Qwen3.8 row); the tables show its exact mode ('STRATA_ROUTE_TAIL_SKIP=0', every routed expert computed, as llama.cpp and tandem.cpp do). Full tables per machine are below.

tandem.cpp is open-source, written primarily in Python, with 108 GitHub stars under the Other license. It was last updated on 2026-11-20.

Key capabilitiesLink to this section

From the project's documentation:

  • Long prompts faster again: layer streaming. A prompt batch of more than 1024 tokens uses every expert of every layer,
  • Warm start (documented now, works with the faster prompts): LLAMA_MOE_HOT_SAVE=hot.txt writes the experts a session
  • Prompts 2–6× faster. With an expert cache, prompts now run in 4096-token batches instead of 512. Each batch has to
  • Store mode: when the RAM store is a few slots short of holding every expert next to the VRAM cache, it now holds all but
  • A big card in a small PC: VRAM and RAM hold every expert. When the VRAM cache plus the RAM store can hold all the
  • Faster first token in store mode. Each expert upload's padding was read from the file, an SSD read per expert.

How it fits a local-AI stackLink to this section

tandem.cpp runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

SourcesLink to this section

Stats from GitHub, 2026-11-21.

Frequently asked

Quick answers to common questions

What is tandem.cpp?

tandem.cpp is a inference-server tool for local AI workloads. llama.cpp patch: faster MoE offload for models that don't fit in VRAM

Is tandem.cpp free and open source?

Yes, tandem.cpp has 108 GitHub stars and is licensed under Other. You can self-host it for free on .

What hardware do I need for tandem.cpp?

The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. tandem.cpp has 108 GitHub stars and an active community.

Does tandem.cpp support GPU acceleration?

tandem.cpp's GPU support depends on your specific setup. Check the documentation for details. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.

What are the best alternatives to tandem.cpp?

Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.

How much does tandem.cpp cost?

tandem.cpp is free-open-source. It is completely free and open source to self-host.

Pairs well with

Complementary tools, models, and hardware