Best local LLMs for 24GB VRAM

The short answer

Top picks for 24 GB: Qwen3.8 27B, A.X-K2, Qwen3.6 27B. 42 benchmarked models fit at Q4.

Fits = Q4_K_M + ~1.8 GB overhead ≤ 24 GB · updated every refresh

Ranked by measured benchmarks

#ModelScoreQ4 sizeHeadroomRun it
1Qwen3.8 27B
70AAArenaGPQA
16 GB+6.2 GBsetup →
2A.X-K2
70GPQA
12 GB+10.2 GBsetup →
3Qwen3.6 27B
67GPQA
16 GB+6.2 GBollama run qwen3.6:27b
4Qwen3.5 35B A3B
65ArenaGPQAMMLU
21 GB+1.2 GBsetup →
5Qwen3.5 9B
61GPQAMMLU
6 GB+16.2 GBsetup →
6K2 Horizon MoVA 36B A4B
59GPQA
22 GB+0.2 GBsetup →
7Gemma 4 26B A4B
57ArenaGPQA
15 GB+7.2 GBsetup →
8Qwen3.6 35B A3B
56GPQA
21 GB+1.2 GBsetup →
9Gemma 4 31B
54AAArenaGPQA
19 GB+3.2 GBsetup →
10EXAONE 4.5 33B
51GPQA
20 GB+2.2 GBsetup →
11Muse Glimmer
46AAArenaGPQA
10 GB+12.2 GBsetup →
12Qwen3.5 4B
42GPQAMMLU
3 GB+19.2 GBsetup →
13NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
41ArenaMMLU
18 GB+4.2 GBsetup →
14Nemotron Cascade 2 30B A3B
41GPQA
18 GB+4.2 GBsetup →
15Qwen3-30B-A3B-Instruct-2507
39Arena
18 GB+4.2 GBsetup →

Also fits, not benchmarked yet

Florence-2-baseQwen3 14BUnlimited-OCRdolphin-2.9.1-yi-1.5-34bSmolLM2-135Mmoondream2DeepSeek-OCRQwen2.5-Coder-14B-Instruct

GPUs with 24 GB–31 GB

Frequently asked

What is the best LLM for 24GB of VRAM?

By measured benchmarks, Qwen3.8 27B, A.X-K2, Qwen3.6 27B lead the models that fit 24 GB at Q4_K_M quantization.

How is "fits" decided?

A model fits when its Q4_K_M weights plus about 1.8 GB of runtime overhead (KV cache, activations) are at or under 24 GB. Long contexts need more; use the VRAM calculator for an exact figure.

How are the models ranked?

Each model's score averages the measured signals it has - Artificial Analysis index, LMArena rating, GPQA, MMLU-Pro - as percentiles across all chat models. Models without any measured benchmark are listed separately, not ranked.