vLLM

vLLM is the serving engine you reach for when Ollama's single-user performance isn't enough. Built at UC Berkeley's Sky Computing Lab, it introduced PagedAttention - a memory management technique that dramatically increases throughput by eliminating the memory waste that plagues other inference engines.

What it is

vLLM is a Python-based LLM serving library that optimizes for throughput and memory efficiency. Its key innovation, PagedAttention, manages the KV cache in fixed-size blocks (like virtual memory paging in operating systems), reducing fragmentation and allowing much larger batch sizes.

Unlike llama.cpp which optimizes for single-user latency, vLLM optimizes for multi-user throughput. If you have a GPU and want to serve a model to many users simultaneously, vLLM is the right choice.

Why this matters

As of June 2026, vLLM has become the standard backend for production local AI deployments:

1. Continuous batching: incoming requests join the next batch automatically, keeping GPU utilization near 100%
2. PagedAttention: eliminates the memory waste that limits concurrent users in other engines
3. Prefix caching: when multiple users share a system prompt, the shared prefix is computed once
4. Speculative decoding: use a small draft model to accelerate the large model by 2-3x

Performance you'll see

All numbers measured with 2k input, 512 output tokens. Source: vLLM benchmark suite.

How it stacks up

What runs on it

• Open WebUI - connect it to vLLM's OpenAI-compatible endpoint
• AnythingLLM - supports vLLM as a provider
• LangChain - native vLLM integration in the Python SDK

Get started

pip install vllm
 
# Serve a model
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-8B \
  --dtype auto \
  --max-model-len 8192
 
# Or use docker
docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
  --model Qwen/Qwen3-8B

What the community says

"vLLM turned our single-GPU Llama serving from 20 req/s to 80 req/s with the same hardware."

• u/ml-infra-eng on r/LocalLLaMA, 445 upvotes

"If you're serving models to more than 5 concurrent users, skip Ollama and go straight to vLLM."

• u/prod-ml-engineer on r/selfhosted, 289 upvotes

When to use something else

• Single-user or desktop use: Ollama or LM Studio are simpler
• Need CPU inference: use llama.cpp instead - vLLM needs CUDA
• Windows user: vLLM has limited Windows support; use TensorRT-LLM or llama.cpp