
NVIDIA RTX 4090
Popular models on the NVIDIA RTX 4090
Fit at Q4_K_M and estimated speed on 24 GB
| Model | Q4 size | Fit | Est. speed |
|---|---|---|---|
| Qwen3.6 35B A3B | 21 GB | Tight | ~238 tok/s |
| dolphin-2.9.1-yi-1.5-34b | 20 GB | Tight | ~31 tok/s |
| Laguna-XS-2.1 | 19 GB | Tight | ~33 tok/s |
| Gemma 4 31B | 19 GB | Tight | ~33 tok/s |
| OTel-2.0-LLM-31B-IT | 19 GB | Tight | ~33 tok/s |
| Qwen3 32B | 19 GB | Tight | ~33 tok/s |
| OTel-2.0-LLM-31B-IT-QLoRA | 18 GB | Fits | ~35 tok/s |
| Qwen3.6 27B | 16 GB | Fits | ~39 tok/s |
| Qwen3.8 27B | 16 GB | Fits | ~39 tok/s |
| Qwen3.8-27B-OBLITERATED | 16 GB | Fits | ~39 tok/s |
| Gemma 4 26B A4B | 15 GB | Fits | ~184 tok/s |
| gpt-oss-20b | 12 GB | Fits | ~208 tok/s |
Estimates: Q4_K_M weights + 1.8 GB overhead; speed at Q4_K_M, scaled from memory bandwidth.
Local inference speed
llama.cpp decode speed (tok/s) - measured
| Workload | tok/s |
|---|---|
| LLaMA 3 8B Q4 | 127.74 |
| LLaMA 3 8B F16 | 54.34 |
Measured - XiongjieDai/GPU-Benchmarks-on-LLM-Inference · updated 2026-11-05
Stacks for the NVIDIA RTX 4090
Complete local setups that run on 24 GB
AI pair programming in your terminal with local models, fully private
Generate photorealistic images locally with node-based AI workflows
Unify multiple local and cloud models behind a single OpenAI-compatible API
A private AI coding assistant in your editor
NVIDIA RTX 4090Link to this section
In 2026, despite the 5090's launch, the RTX 4090 remains the most popular GPU on r/LocalLLaMA - and for good reason. 24 GB VRAM runs every model in the "sweet spot" range (Qwen3-30B, Mistral Small 3, Gemma 3 27B, DeepSeek Coder V3) at Q4_K_M or Q5_K_M with room for serious context. The used price has crashed to $1200-1500 since the 5090 launched.
Quick verdictLink to this section
The 4090 is the answer when:
- You want the cheapest GPU that runs 30B models well
- You don't need 70B in a single GPU (use Llama 3.3 70B Q3 if you must, but it's tight)
- You're buying once and not upgrading for 3-5 years
Get a 5090 if you specifically want comfortable 70B inference. Otherwise the 4090 is 80% of the experience at 60% of the price.
Real-world AI inferenceLink to this section
| Model | Tokens/sec | Notes |
|---|---|---|
| Qwen3-30B Q4_K_M | ~25 tok/s | Sweet spot |
| Llama 3.3 70B Q3_K_M | ~12 tok/s | Tight fit, usable |
| Mistral Small 3 Q5_K_M | ~42 tok/s | Plenty of room |
| ComfyUI SDXL (1024x1024) | ~10s/image | Excellent for image gen |
| ComfyUI Flux Dev | ~32s/image | Reasonable |
Spec breakdownLink to this section
- VRAM: 24 GB GDDR6X
- Memory bandwidth: 1008 GB/s
- TDP: 450 W
- PCIe: 4.0 ×16
- Slot count: 3-slot
- Power: 850W PSU minimum, 1000W comfortable
Best models that fitLink to this section
- Q4_K_M: Qwen3-30B (18GB), Mistral Small 3 (14GB) - comfortable
- Q5_K_M: Qwen3-30B (22GB) at smaller context
- Q8_0: 13-14B models like Qwen3-14B
- Llama 3.3 70B at Q3: tight but works, ~12 tok/s
Used market realityLink to this section
The 5090 launch crashed used 4090 prices. Sweet spots on eBay:
- $1200-1400: well-treated cards from non-mining sources (last 18 months)
- $1000-1200: heavier-use cards, verify fans/thermals
- Under $1000: probably ex-mining, avoid unless you can verify
ALWAYS run the seller's serial through NVIDIA warranty check.
Honest alternativesLink to this section
- RTX 5090 new ($2000-2200): if 70B is mandatory
- Dual RTX 3090 used build (~$1400-1800): same 48GB VRAM, worse power efficiency
- Mac Studio M4 Ultra ($4k+): bigger VRAM ceiling, slower per-token
Frequently asked
Quick answers to common questions
How much VRAM does the NVIDIA RTX 4090 have?
The NVIDIA RTX 4090 has 24 GB of VRAM with 1008 GB/s memory bandwidth. MSRP was $1,599.
What local AI models can run on the NVIDIA RTX 4090?
The NVIDIA RTX 4090 with 24 GB VRAM can run many models depending on quantization. Models up to ~36B params may fit at Q4_K_M. Use our VRAM calculator to check specific models.
Is the NVIDIA RTX 4090 good for local AI inference?
NVIDIA RTX 4090 is best for llm-inference, image-gen, comfyui. With ample VRAM it handles most open models well.
Where can I buy the NVIDIA RTX 4090?
Check our buy links above for the best current prices on Amazon, Newegg, and B&H. Prices vary by retailer and availability.
How does the NVIDIA RTX 4090 compare to other GPUs?
NVIDIA RTX 4090 has 24 GB VRAM and 1008 GB/s bandwidth. This puts it in the high-end category, suitable for most open models. Browse our hardware directory for side-by-side comparisons.
Is the NVIDIA RTX 4090 worth buying right now?
The current price is $1599. The price is at or above MSRP. Consider waiting for sales events like Prime Day or Black Friday.
What power supply do I need for the NVIDIA RTX 4090?
The NVIDIA RTX 4090 has a TDP of 450W. This requires a high-wattage PSU (850W+ recommended). Always check the manufacturer's recommendations for your specific build.
Nearby options
Similar hardware and models that fit