NVIDIA Jetson AGX Thor (T5000)
Local inference speed
llama.cpp decode speed (tok/s) - estimated
| Workload | tok/s |
|---|---|
| LLaMA 3 8B Q4 | 32 |
| LLaMA 3 8B F16 | 14.5 |
| LLaMA 3 70B Q4 | 3.8 |
Estimated - Estimated from memory bandwidth (llama.cpp decode) · updated 2026-10-31
Stacks for the NVIDIA Jetson AGX Thor (T5000)
Complete local setups that run on 128 GB
Run Qwen3.6-27B with vision, 256K context, and hybrid thinking locally
AI pair programming in your terminal with local models, fully private
A private AI assistant on Windows with document RAG - no CLI, no Docker, all desktop GUI
A private AI coding assistant in your editor
Prototype and deploy multi-agent RAG applications visually with local models
NVIDIA Jetson AGX Thor (T5000)
The NVIDIA Jetson AGX Thor (T5000) is a GPU built on the Blackwell architecture, released 2025-08-27. For running AI locally, the numbers that matter are its 128 GB of LPDDR5X and 273.2 GB/s of memory bandwidth. VRAM decides which models fit at all; bandwidth sets how fast they generate text.
What you can run on 128 GB
At Q4_K_M quantization (the usual local default), 128 GB holds models up to roughly 203B parameters, leaving headroom for context. On this card you can run, among others:
- Qwen3.8-Flash-Next - 180B parameters
- MiMo-V2.6-Pro - 159.4B parameters
- Inkling Small - 156B parameters
- Qwen3.5 122B A10B - 125.1B parameters
- NVIDIA-Nemotron-3-Super-120B-A12B-BF16 - 123.6B parameters
Larger models need a higher-VRAM card, a second GPU, or CPU offload (which is much slower). Check any specific model with the VRAM calculator, or see the full picture on what can I run.
Local LLM speed (LLaMA 3, llama.cpp)
Single-stream token-generation throughput - estimated from memory bandwidth:
| Model (quant) | Speed on NVIDIA Jetson AGX Thor (T5000) |
|---|---|
| Llama 3 8B (Q4_K_M) | 32 tok/s |
| Llama 3 8B (F16) | 14.5 tok/s |
| Llama 3 70B (Q4_K_M) | 3.8 tok/s |
Because decode is memory-bandwidth bound, the 273.2 GB/s figure is the best single predictor of chat speed on this card. Estimates are calibrated against measured RTX-40-series cards and are typically within ~15%.
Memory and power
- VRAM: 128 GB LPDDR5X
- Bandwidth: 273.2 GB/s
- TDP: 40 W
Quantization and context
Quantization trades a little quality for a lot of VRAM. On 128 GB you can fit roughly a 203B model at Q4_K_M, about a 110B model at the higher-quality Q8, or a smaller model at full FP16. Longer context windows also consume VRAM (the KV cache grows with context length), so leave a few GB of headroom if you plan to use large prompts or many concurrent requests. For most chat and coding use, Q4_K_M on this card is the sweet spot between speed, quality, and the 128 GB budget.
How it compares
Similar cards for local AI, by VRAM and 8B-Q4 speed:
| GPU | VRAM | Bandwidth | Llama 3 8B Q4 |
|---|---|---|---|
| NVIDIA Jetson AGX Thor (T5000) | 128 GB | 273.2 GB/s | 32 tok/s |
| NVIDIA DGX Spark (GB10) | 128 GB | 273.2 GB/s | 32 tok/s |
| RTX PRO 6000 Blackwell | 96 GB | 1790 GB/s | 209.5 tok/s |
| RTX PRO 6000 Blackwell Max-Q | 96 GB | 1790 GB/s | 209.5 tok/s |
Bottom line
The NVIDIA Jetson AGX Thor (T5000) is best for llm-inference, image-gen, comfyui. With 24 GB+ it comfortably handles most open models, including 30B-class at Q4. If you need more, compare with NVIDIA DGX Spark (GB10) and RTX PRO 6000 Blackwell.
Sources
- Specifications: RightNow GPU Database (TechPowerUp data)
- Benchmarks: GPU-Benchmarks-on-LLM-Inference (basis for the bandwidth estimate)
Specs and benchmarks last checked 2026-10-30. Verify current pricing before buying.
Frequently asked
Quick answers to common questions
How much VRAM does the NVIDIA Jetson AGX Thor (T5000) have?
The NVIDIA Jetson AGX Thor (T5000) has 128 GB of VRAM with 273.2 GB/s memory bandwidth.
What local AI models can run on the NVIDIA Jetson AGX Thor (T5000)?
The NVIDIA Jetson AGX Thor (T5000) with 128 GB VRAM can run many models depending on quantization. Models up to ~196B params may fit at Q4_K_M. Use our VRAM calculator to check specific models.
Is the NVIDIA Jetson AGX Thor (T5000) good for local AI inference?
NVIDIA Jetson AGX Thor (T5000) is best for llm-inference, image-gen, comfyui. With ample VRAM it handles most open models well.
Where can I buy the NVIDIA Jetson AGX Thor (T5000)?
Check our buy links above for the best current prices on Amazon, Newegg, and B&H. Prices vary by retailer and availability.
How does the NVIDIA Jetson AGX Thor (T5000) compare to other GPUs?
NVIDIA Jetson AGX Thor (T5000) has 128 GB VRAM and 273.2 GB/s bandwidth. This puts it in the high-end category, suitable for most open models. Browse our hardware directory for side-by-side comparisons.
What power supply do I need for the NVIDIA Jetson AGX Thor (T5000)?
The NVIDIA Jetson AGX Thor (T5000) has a TDP of 40W. A standard quality PSU of 650W+ should suffice. Always check the manufacturer's recommendations for your specific build.
Nearby options
Similar hardware and models that fit