dynamo

dynamo is an image-generation tool - A Datacenter Scale Distributed Inference Serving Framework.

Overview

The open-source, datacenter-scale inference stack. Dynamo is the orchestration layer above inference engines — it doesn't replace SGLang, TensorRT-LLM, or vLLM, it turns them into a coordinated multi-node inference system. Disaggregated serving, intelligent routing, multi-tier KV caching, and automatic scaling work together to maximize throughput and minimize latency for LLM, reasoning, multimodal, and video generation workloads.

If you're running a single model on a single GPU, your inference engine alone is probably sufficient.

Most inference engines optimize a single GPU or a single node. Dynamo is the orchestration layer above them — it turns a cluster of GPUs into a coordinated inference system.

• Zero-config deploy (DGDR) (beta): Specify model, HW, and SLA in one YAML — AIConfigurator auto-profiles the workload, Planner optimizes the topology, and Dynamo deploys - Agentic inference: Per-request hints for priority, expected output length, and speculative prefill, plus session metadata for tracing and SGLang subagent KV isolation. LangChain + NeMo Agent Toolkit integrations - Multimodal E/P/D: Disaggregated encode/prefill/decode with embedding cache — 30% faster TTFT on image workloads - Video generation: Native FastVideo + SGLang Diffusion support — real-time 1080p on single B200 - K8s Inference Gateway plugin: KV-aware routing inside the standard Kubernetes gateway - Storage-tier KV offload: S3/Azure blob support + global KV events for cluster-wide cache visibility

dynamo is open-source, written primarily in Rust, with 7,398 GitHub stars under the Other license. The latest release is v1.2.1 (2026-06-13).

Key capabilities

From the project's documentation:

• You're serving LLMs across multiple GPUs or nodes and need to coordinate them
• You want KV-aware routing to avoid redundant prefill computation
• You need to independently scale prefill and decode (disaggregated serving)
• You want automatic scaling that meets latency SLAs at minimum total cost of ownership (TCO)
• You need fast cold-starts when spinning up new replicas
• Multimodal E/P/D: Disaggregated encode/prefill/decode with embedding cache — 30% faster TTFT on image workloads

Install

A quick way to get started (always check the official docs for the latest):

pip install --prerelease=allow "ai-dynamo[sglang]" # or [vllm]

How it fits a local-AI stack

dynamo runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related image-generation tools in the directory:

• Stable Diffusion WebUI (AUTOMATIC1111)
• ComfyUI
• Fooocus
• InvokeAI

Sources

• Source code & docs: ai-dynamo/dynamo
• Official website: https://docs.nvidia.com/dynamo/latest

Stats from GitHub, 2026-07-02.