PowerInfer social preview
inference-server9,885MIT

PowerInfer

High-speed Large Language Model Serving for Local Deployment

Updated Nov 14, 2026
Platforms
windows
Pricing
free-open-source
Status
active
License
MIT

What it does

Core capabilities at a glance

  • Large Language Models
  • Llama
  • LLM Inference
  • Local Inference

Deep dive

The full breakdown - performance, comparisons, and setup

PowerInferLink to this section

PowerInfer is a local inference server - High-speed Large Language Model Serving for Local Deployment.

OverviewLink to this section

PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device.

PowerInfer is open-source, written primarily in C++, with 9,885 GitHub stars under the MIT license. It was last updated on 2026-05-11.

Key capabilitiesLink to this section

From the project's documentation:

  • [2024/5/17] We now provide support for AMD devices with ROCm.
  • [2024/1/11] We supported Windows with GPU inference!
  • [2023/12/24] We released an online gradio demo for Falcon(ReLU)-40B-FP16!
  • [2023/12/19] We officially released PowerInfer!
  • Easy Integration: Compatible with popular ReLU-sparse models.
  • x86-64 CPUs with AVX2 instructions, with or without NVIDIA GPUs, under Linux.

InstallLink to this section

A quick way to get started (always check the official docs for the latest):

pip install -r requirements.txt # install Python helpers' dependencies

How it fits a local-AI stackLink to this section

PowerInfer runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

SourcesLink to this section

Stats from GitHub, 2026-06-08.

From the community

What users are saying

“> PowerInfer’s source code is publicly available at https://github.com/SJTU-IPADS/PowerInfer --- Just curious - PowerInfer seems to market itself by running very large models (40B, 70B) on something like a 4090.”

“PowerInfer is a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation.”

“From their GitHub[0]: > Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU.”

Frequently asked

Quick answers to common questions

What is PowerInfer?

PowerInfer is a inference-server tool for local AI workloads. High-speed Large Language Model Serving for Local Deployment

Is PowerInfer free and open source?

Yes, PowerInfer has 9,885 GitHub stars and is licensed under MIT. You can self-host it for free on windows.

What platforms does PowerInfer support?

PowerInfer runs on windows.

What hardware do I need for PowerInfer?

The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. PowerInfer has 9,885 GitHub stars and an active community.

Does PowerInfer support GPU acceleration?

PowerInfer supports GPU acceleration via CUDA, Metal, or Vulkan depending on your platform. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.

What are the best alternatives to PowerInfer?

Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.

How much does PowerInfer cost?

PowerInfer is free-open-source. It is completely free and open source to self-host.

Pairs well with

Complementary tools, models, and hardware