OpenLLM

OpenLLM is a fine-tuning toolkit - Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

Overview

OpenLLM allows developers to run any open-source LLMs (Llama 3.3, Qwen2.5, Phi3 and more) or custom models as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Docker, Kubernetes, and BentoCloud.

Run the following commands to install OpenLLM and explore it interactively.

OpenLLM supports a wide range of state-of-the-art open-source LLMs. You can also add a model repository to run custom models with OpenLLM.

To start an LLM server locally, use the 'openllm serve' command and specify the model version.

The server will be accessible at http://localhost:3000, providing OpenAI-compatible APIs for interaction. You can call the endpoints with different frameworks and tools that support OpenAI-compatible APIs. Typically, you may need to specify the following:

• The API host address: By default, the LLM is hosted at http://localhost:3000. - The model name: The name can be different depending on the tool you use. - The API key: The API key used for client authentication. This is optional.

OpenLLM provides a chat UI at the '/chat' endpoint for the launched LLM server at http://localhost:3000/chat.

To start a chat conversation in the CLI, use the 'openllm run' command and specify the model version.

OpenLLM is open-source, written primarily in Python, with 12,351 GitHub stars under the Apache 2.0 license. The latest release is v0.6.30 (2025-04-21).

Key capabilities

From the project's documentation:

• The API host address: By default, the LLM is hosted at http://localhost:3000.
• The model name: The name can be different depending on the tool you use.
• The API key: The API key used for client authentication. This is optional.
• Repost a bug by creating a GitHub issue.
• Check out the Developer Guide to learn more.
• bentoml/bentoml for production level model serving

Install

A quick way to get started (always check the official docs for the latest):

pip install openllm # or pip3 install openllm

How it fits a local-AI stack

OpenLLM runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related fine-tuning toolkits in the directory:

• Axolotl
• LitGPT
• LLaMA-Factory
• Torchtune

Sources

• Source code & docs: bentoml/OpenLLM
• Official website: https://bentoml.com

Stats from GitHub, 2026-06-08.