llama-modchip social preview
inference-server536Apache 2.0

llama-modchip

🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs). SwiGLU-H2, Stockfi…

Updated Nov 9, 2026
Website GitHub Docs
Platforms
macos, linux
Pricing
free-open-source
Status
active
License
Apache 2.0

What it does

Core capabilities at a glance

  • AI Agents
  • Apple Silicon
  • Avx512
  • Claude Code
  • CPU Inference
  • Deepseek
  • Deepseek V4
  • Gguf

Deep dive

The full breakdown - performance, comparisons, and setup

llama-modchipLink to this section

llama-modchip is a local inference server - 🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs). SwiGLU-H2, Stockfish-PVS L3 & 13ms Claude Code restore.

OverviewLink to this section

8-Channel Workstation (Threadripper) 128 GB+ DeepSeek-284B / Nemotron-30B 111.62 tok/s (30B) 8-CCD AVX-512BW _1x8 register tiling saturates 8-channel DDR5 bandwidth

'llama-modchip' is a set of 'ggml'/'llama.cpp' kernel patches, quantization transcoders, and runtime tools designed to run sparse Mixture-of-Experts (MoE) checkpoints entirely in system RAM on x86_64 ('AVX-512' / 'AVX2') and ARM64 ('NEON' / 'SVE' / Apple Metal) processors:

llama-modchip is open-source, written primarily in Python, with 536 GitHub stars under the Apache 2.0 license. The latest release is v2.4.0 (2026-11-04).

How it fits a local-AI stackLink to this section

llama-modchip runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

SourcesLink to this section

Stats from GitHub, 2026-11-09.

Frequently asked

Quick answers to common questions

What is llama-modchip?

llama-modchip is a inference-server tool for local AI workloads. 🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs). SwiGLU-H2, Stockfi…

Is llama-modchip free and open source?

Yes, llama-modchip has 536 GitHub stars and is licensed under Apache 2.0. You can self-host it for free on macos, linux.

What platforms does llama-modchip support?

llama-modchip runs on macos, linux.

What hardware do I need for llama-modchip?

The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. llama-modchip has 536 GitHub stars and an active community.

Does llama-modchip support GPU acceleration?

llama-modchip supports GPU acceleration via CUDA, Metal, or Vulkan depending on your platform. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.

What are the best alternatives to llama-modchip?

Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.

How much does llama-modchip cost?

llama-modchip is free-open-source. It is completely free and open source to self-host.

Pairs well with

Complementary tools, models, and hardware