lorax

lorax is a fine-tuning toolkit - Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs.

Overview

LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.

• 🚅 Dynamic Adapter Loading: include any fine-tuned LoRA adapter from HuggingFace, Predibase, or any filesystem in your request, it will be loaded just-in-time without blocking concurrent requests. Merge adapters per request to instantly create powerful ensembles. - 🏋️‍♀️ Heterogeneous Continuous Batching: packs requests for different adapters together into the same batch, keeping latency and throughput nearly constant with the number of concurrent adapters. - 🧁 Adapter Exchange Scheduling: asynchronously prefetches and offloads adapters between GPU and CPU memory, schedules request batching to optimize the aggregate throughput of the system. - 👬 Optimized Inference: high throughput and low latency optimizations including tensor parallelism, pre-compiled CUDA kernels (flash-attention, paged attention, SGMV), quantization, token streaming. - 🚢 Ready for Production prebuilt Docker images, Helm charts for Kubernetes, Prometheus metrics, and distributed tracing with Open Telemetry. OpenAI compatible API supporting multi-turn chat conversations. Private adapters through per-request tenant isolation. Structured Output (JSON mode). - 🤯 Free for Commercial Use: Apache 2.0 License. Enough said 😎.

• Base Model: pretrained large model shared across all adapters. - Adapter: task-specific adapter weights dynamically loaded per request.

lorax is open-source, written primarily in Python, with 3,790 GitHub stars under the Apache 2.0 license. The latest release is lorax-0.4.0 (2025-01-13).

Key capabilities

From the project's documentation:

• 📖 Table of contents
• Prompt via REST API
• Prompt via Python Client
• Chat via OpenAI API
• 🤯 Free for Commercial Use: Apache 2.0 License. Enough said 😎.
• Base Model: pretrained large model shared across all adapters.

Install

A quick way to get started (always check the official docs for the latest):

pip install lorax-client

How it fits a local-AI stack

lorax runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related fine-tuning toolkits in the directory:

• Axolotl
• LitGPT
• LLaMA-Factory
• Torchtune

Sources

• Source code & docs: predibase/lorax
• Official website: https://loraexchange.ai

Stats from GitHub, 2026-06-08.