TurboTransformers
a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
What it does
Core capabilities at a glance
- Albert
- Bert
- Decoder
- Gpt2
- GPU
- Huggingface Transformers
- Inference
- Machine Translation
Deep dive
The full breakdown - performance, comparisons, and setup
TurboTransformersLink to this section
TurboTransformers is a local inference server - a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
OverviewLink to this section
Make transformers serving fast by adding a turbo to your inference engine!
The WeChat AI open-sourced TurboTransformers with the following characteristics.
- Supporting both Transformers Encoder and Decoder. 3. Supports Variable Length inputs. No time-consuming offline tuning is required. You can change batch size and sequence length at real-time. 3. Excellent CPU / GPU performance. 4. Perfect Usibility. TurboTransformers supports python and C++ APIs. 5. Smart Batching. Minimize zero-padding overhead for a batch of requests of different lengths. It can be used as a plugin for pytorch. Tthe end-to-end acceleration is obtained by adding a few lines of python code.
TurboTransformers has been applied to multiple online BERT service scenarios in Tencent. For example, It brings 1.88x acceleration to the WeChat FAQ service, 2.11x acceleration to the public cloud sentiment analysis service, and 13.6x acceleration to the QQ recommendation system. Moreover, it has already been applied to build services such as Chitchating, Searching, and Recommendation.
The following table is a comparison of TurboTransformers and related work.
- BERT [Python] [C++] * ALBERT [Python] * Roberta [Python] * Transformer Decoder [Python] * GPT2 [Python]
Note that the building scripts only apply to specific OS and software (Pytorch, OpenNMT, transformers, etc.) versions. Please adjust them according to your needs.
- Run benchmark (optional) in docker, compare with pytorch, torch-JIT, onnxruntime
TurboTransformers is open-source, written primarily in C++, with 1,551 GitHub stars under the Other license. The latest release is v0.5.1 (2020-11-25).
InstallLink to this section
A quick way to get started (always check the official docs for the latest):
docker run -it --rm --name=turbort -v $PWD:/workspace your_image_name /bin/bashHow it fits a local-AI stackLink to this section
TurboTransformers runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:
SourcesLink to this section
- Source code & docs: Tencent/TurboTransformers
Stats from GitHub, 2026-10-30.
Frequently asked
Quick answers to common questions
What is TurboTransformers?
TurboTransformers is a inference-server tool for local AI workloads. a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
Is TurboTransformers free and open source?
Yes, TurboTransformers has 1,551 GitHub stars and is licensed under Other. You can self-host it for free on .
What hardware do I need for TurboTransformers?
The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. TurboTransformers has 1,551 GitHub stars and an active community.
Does TurboTransformers support GPU acceleration?
TurboTransformers's GPU support depends on your specific setup. Check the documentation for details. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.
What are the best alternatives to TurboTransformers?
Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.
How much does TurboTransformers cost?
TurboTransformers is free-open-source. It is completely free and open source to self-host.
Pairs well with
Complementary tools, models, and hardware