CTranslate2

CTranslate2 is a local inference server - Fast inference engine for Transformer models.

Overview

CTranslate2 is a C++ and Python library for efficient inference with Transformer models.

The project implements a custom runtime that applies many performance optimization techniques such as weights quantization, layers fusion, batch reordering, etc., to accelerate and reduce the memory usage of Transformer models on CPU and GPU.

Compatible models should be first converted into an optimized model format. The library includes converters for multiple frameworks:

The project is production-oriented and comes with backward compatibility guarantees, but it also includes experimental features related to model compression and inference acceleration.

CTranslate2 is open-source, written primarily in C++, with 4,717 GitHub stars under the MIT license. The latest release is v4.8.3 (2026-10-13).

Key capabilities

From the project's documentation:

• Decoder-only models: GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, MPT, Llama, Mistral, Gemma, CodeGen, GPTBigCode, Falcon, Qwen2
• Encoder-only models: BERT, DistilBERT, XLM-RoBERTa
• Lightweight on diskQuantization can make the models 4 times smaller on disk with minimal accuracy loss.
• New Model Support: Help us implement more Transformer architectures.
• Performance: Propose optimizations for CPU or GPU kernels.
• Bug Reports: Open an issue if you find something not working as expected.

Install

A quick way to get started (always check the official docs for the latest):

pip install ctranslate2

How it fits a local-AI stack

CTranslate2 runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

• "BentoML"
• "candle-vllm"
• ExLlamaV2
• "flashinfer"

Sources

• Source code & docs: OpenNMT/CTranslate2
• Official website: https://opennmt.net/CTranslate2

Stats from GitHub, 2026-10-24.