vllm-mlx

vllm-mlxLink to this section

vllm-mlx is a speech (TTS/STT) tool - High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

OverviewLink to this section

Continuous batching + OpenAI + Anthropic APIs in one server. Native Apple Silicon inference.

A vLLM-style inference server for Apple Silicon Macs. Unlike 'Ollama' or 'mlx-lm' used directly, it ships continuous batching, paged KV cache, prefix caching, and SSD-tiered cache, and exposes both OpenAI '/v1/*' and Anthropic '/v1/messages' from a single process. Run LLMs, vision models, audio, and embeddings on Metal with unified memory, no conversion step.

• Continuous batching: high throughput for concurrent requests - Paged KV cache: memory-efficient with prefix sharing - SSD-tiered KV cache: spill prefix cache to disk for long-context agents ('--ssd-cache-dir') - Warm prompts: preload popular prefixes at startup ('--warm-prompts') for 1.3-2.25x TTFT - Prefix cache: trie-based, shared across requests

See docs/benchmarks/ for continuous-batching results, KV-cache quantization (4-bit / 8-bit / fp16), and MoE top-k sweeps.

The built-in MLX reranker forward path supports standard BERT/XLM-RoBERTa sequence-classification weights with 'gelu', 'gelu_new'/'gelu_fast', 'relu', or 'silu'/'swish' 'hidden_act' values. Other activations fail explicitly so custom reranker architectures can add a dedicated adapter instead of silently using the wrong activation.

Bug fixes, perf work, docs, and benchmarks on different Apple Silicon chips all welcome. See the Contributing Guide.

vllm-mlx is open-source, written primarily in Python, with 1,651 GitHub stars under the Apache 2.0 license. The latest release is v0.4.1 (2026-08-12).

Key capabilitiesLink to this section

From the project's documentation:

• OpenAI-compatible: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank, /v1/responses
• Anthropic-compatible: /v1/messages (streaming, tool use, system prompts)
• MCP Tool Calling: 12 parsers (OpenAI, Anthropic, Gemini, Qwen, DeepSeek, Gemma, and more)
• Structured output: JSON Schema via response_format (lm-format-enforcer)
• Continuous batching: high throughput for concurrent requests
• Paged KV cache: memory-efficient with prefix sharing

InstallLink to this section

A quick way to get started (always check the official docs for the latest):

pip install vllm-mlx

How it fits a local-AI stackLink to this section

vllm-mlx runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related speech (TTS/STT) tools in the directory:

• "cactus"
• "claude-code-local"
• "dograh"
• "DreamServer"

SourcesLink to this section

• Source code & docs: waybarrios/vllm-mlx
• Official website: https://pypi.org/project/vllm-mlx/

Stats from GitHub, 2026-08-13.