kindi

kindiLink to this section

kindi is a local inference server - Kindi 0.6: fast local LLM inference on low-cost GPUs and CPUs. DeepSeek-V4.1-Flash (552B) at 18.7 tok/s on one RTX 3060 12 GB with a dual-Xeon DDR4 server. Ternary Bonsai 2B-27B, AH-Nanbeige4.3-33B, Nemotron 3.5 Lightning 30B-A3B. NVIDIA from the Quadro K420 to the RTX 50 series, AMD Radeon graphics, Windows and Linux. OpenAI-compatible server.

OverviewLink to this section

Kindi is a custom inference engine for language models on low-cost GPUs. It runs Prism ML's ternary Bonsai models (2B, 4B, 9B and 27B), AH-Nanbeige4.3-33B, Nemotron 3.5 Lightning 30B-A3B and DeepSeek-V4.1-Flash. Kindi supports NVIDIA cards from the 1 GB Quadro K420 (Kepler) to the RTX 50 series and the CMP 170HX. It also supports AMD Radeon 780M, 840M and 8060S graphics, and x86-64 CPUs. The OpenAI-compatible server has a prompt cache, tool calls, a thinking mode and speculative decoding. With the exact head, the greedy output is the same as the output of llama.cpp for the same model file, token for token. Version 0.6.0 adds two mixture-of-experts models that keep experts in host memory, the DSpine drafter for the 27B, and a one-command installer for Windows.

Kindi and llama.cpp on the same card, with the same model file and the same context. Tokens per second, one stream, greedy. The method and the other cards: docs/BENCHMARKS.md.

On the 8 GB cards, the Lightning model does not fit on the card. Both engines keep some experts in host memory (llama.cpp: '-ncmoe 9' on the GTX 1080 and '-ncmoe 12' on the RTX 2070 SUPER). On these two cards the write test uses a real text as the prompt.

kindi is open-source, written primarily in Python, with 158 GitHub stars under the AGPL-3.0 license. The latest release is v0.6.0 (2026-11-12).

Key capabilitiesLink to this section

From the project's documentation:

• The 27B with DSpine. On an RTX 3080 at 190 W and a context of 32,768 tokens, the 27B writes 115.5 tokens/s
• DeepSeek-V4.1-Flash (552B). On one RTX 3060 12 GB, a Quadro M4000 for the most used experts and two Xeon
• CHANGELOG.md: what changed in each version
• docs/BENCHMARKS.md and docs/CARDS.md: the measurements and the settings
• WHY.md: the one-page summary
• First steps after the install

InstallLink to this section

A quick way to get started (always check the official docs for the latest):

curl -fsSL https://amirhusain.com/kindi/install.sh | bash

How it fits a local-AI stackLink to this section

kindi runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

• "airllm"
• "Atomic-Chat"
• "backburner"
• "BentoML"

SourcesLink to this section

• Source code & docs: ahtx/kindi
• Official website: https://amirhusain.com/software/kindi/

Stats from GitHub, 2026-11-13.