server

server is a local inference server - The Triton Inference Server provides an optimized cloud and edge inferencing solution.

Overview

Triton Inference Server is an open source inference serving software that streamlines AI inferencing. Triton enables teams to deploy any AI model from multiple deep learning and machine learning frameworks, including TensorRT, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton Inference Server supports inference across cloud, data center, edge and embedded devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton Inference Server delivers optimized performance for many query types, including real time, batched, ensembles and audio/video streaming. Triton inference Server is part of NVIDIA AI Enterprise, a software platform that accelerates the data science pipeline and streamlines the development and deployment of production AI.

• Supports multiple deep learning frameworks - Supports multiple machine learning frameworks - Concurrent model execution - Dynamic batching - Sequence batching and implicit state management for stateful models - Provides Backend API that allows adding custom backends and pre/post processing operations - Supports writing custom backends in python, a.k.a. Python-based backends. - Model pipelines using Ensembling or Business Logic Scripting (BLS) - HTTP/REST and GRPC inference protocols based on the community developed KServe protocol - A C API and Java API allow Triton to link directly into your application for edge and other in-process use cases - Metrics indicating GPU utilization, server throughput, server latency, and more

server is open-source, written primarily in Python, with 11,101 GitHub stars under the BSD-3-Clause license. The latest release is v2.73.0 (2026-09-30).

Key capabilities

From the project's documentation:

• [Supports multiple deep learning
• [Supports multiple machine learning
• Provides Backend API that
• Supports writing custom backends in python, a.k.a.
• [HTTP/REST and GRPC inference
• A C API and

Install

A quick way to get started (always check the official docs for the latest):

docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:26.09-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx

How it fits a local-AI stack

server runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

• "BentoML"
• "candle-vllm"
• ExLlamaV2
• "flashinfer"

Sources

• Source code & docs: triton-inference-server/server
• Official website: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html

Stats from GitHub, 2026-10-22.