whichllm

whichllm is a local inference server - Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.

Overview

Find the best local LLM that actually runs on your hardware.

Auto-detects your GPU/CPU/RAM and ranks the top models from HuggingFace that fit your system.

After install, run 'whichllm' directly. For one-off runs, replace 'whichllm' with 'uvx whichllm@latest'.

The 32B model fits your card fine — whichllm still ranks the 27B #1, because it scores higher on real benchmarks and is a newer generation. A size-only "what fits?" tool would hand you the bigger one. That gap is the whole point of whichllm. (Note #3: a MoE model at 102 t/s — speed is ranked on active params, quality on total.)

Fitting a model into your VRAM is the easy part. The hard part is knowing which of the models that fit is actually the best — and that is what whichllm is built to get right.

whichllm is open-source, written primarily in Python, with 3,122 GitHub stars under the MIT license. The latest release is v0.5.8 (2026-06-05).

Key capabilities

From the project's documentation:

• Evidence-based ranking, not a size heuristic — The top pick is
• Recency-aware — Stale leaderboards are demoted along each model's
• Evidence-graded and guarded — Every score is tagged
• Architecture-aware estimates — VRAM = weights + GQA KV cache +
• One command, scriptable — whichllm prints the answer; add
• Live data — Models fetched directly from the HuggingFace API, with

Install

A quick way to get started (always check the official docs for the latest):

pip install whichllm

How it fits a local-AI stack

whichllm runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

• ExLlamaV2
• llama.cpp
• LocalAI
• Ollama

Sources

• Source code & docs: Andyyyy64/whichllm

Stats from GitHub, 2026-06-08.