What it does
Core capabilities at a glance
- FP8, INT4, INT8 quantization with minimal quality loss
- In-flight batching for maximum throughput
- Multi-GPU and multi-node tensor parallelism
- PagedAttention and KV cache management
- OpenAI-compatible API server
- Optimized for Llama, Falcon, Mistral, Gemma, Qwen
Deep dive
The full breakdown - performance, comparisons, and setup
TensorRT-LLM
TensorRT-LLM is NVIDIA's official LLM inference optimization library. It delivers the highest possible throughput on NVIDIA hardware through aggressive kernel optimization and quantization.
What it is
TRT-LLM is an open-source library that optimizes LLM inference on NVIDIA GPUs. It compiles models into optimized TensorRT engines with FP8, INT4, and INT8 quantization, in-flight batching, and multi-GPU parallelism.
Get started
pip install tensorrt_llm
# Build engine
trtllm-build --model_dir Qwen/Qwen3-8B \
--dtype float16 --use_gpt_attention_plugin float16
# Run server
python examples/launch_triton_server.py \
--model_repo trtllm_qwen_engineWhen to use something else
From the community
What users are saying
“From the linked article: "TensorRT-LLM consists of the TensorRT deep learning compiler and includes optimized kernels, pre- and post-processing steps, and multi-GPU/multi-node communication primitives for groundbreaking performance on NVIDIA GPUs.”
“There’s a percentage of users that do care about token generation speed as they chain multiple API calls. The performance is all thanks to TensorRT-LLM, Mystic takes care of the engineering of getting a scalable endpoint out of it, i.e, not having to manage your infra.”
“"NVIDIA TensorRT-LLM is now available in early access and soon will be integrated into the NVIDIA NeMo framework—part of NVIDIA AI Enterprise, an enterprise-grade AI software platform with security, stability, manageability, and support.”
Frequently asked
Quick answers to common questions
What is TensorRT-LLM?
TensorRT-LLM is a inference-server tool for local AI workloads. NVIDIA's optimized LLM inference engine delivering maximum performance on NVIDIA GPUs.
Is TensorRT-LLM free and open source?
Yes, TensorRT-LLM has 14,851 GitHub stars and is licensed under Apache-2.0. You can self-host it for free on linux.
What platforms does TensorRT-LLM support?
TensorRT-LLM runs on linux.
What hardware do I need for TensorRT-LLM?
The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. TensorRT-LLM has 14,851 GitHub stars and an active community.
Does TensorRT-LLM support GPU acceleration?
TensorRT-LLM supports GPU acceleration via CUDA, Metal, or Vulkan depending on your platform. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.
What are the best alternatives to TensorRT-LLM?
Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.
How much does TensorRT-LLM cost?
TensorRT-LLM is free-open-source. It is completely free and open source to self-host.
Pairs well with
Complementary tools, models, and hardware