PowerInfer social preview
inference-server9,540MIT

PowerInfer

High-speed Large Language Model Serving for Local Deployment

Updated Jun 8, 2026
GitHub Docs
Platforms
windows
Pricing
free-open-source
Status
active
License
MIT

What it does

Core capabilities at a glance

  • Large Language Models
  • Llama
  • LLM Inference
  • Local Inference

Deep dive

The full breakdown - performance, comparisons, and setup

PowerInfer

PowerInfer is a local inference server - High-speed Large Language Model Serving for Local Deployment.

Overview

PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device.

PowerInfer is open-source, written primarily in C++, with 9,540 GitHub stars under the MIT license. It was last updated on 2026-05-11.

Key capabilities

From the project's documentation:

  • [2024/5/17] We now provide support for AMD devices with ROCm.
  • [2024/1/11] We supported Windows with GPU inference!
  • [2023/12/24] We released an online gradio demo for Falcon(ReLU)-40B-FP16!
  • [2023/12/19] We officially released PowerInfer!
  • Easy Integration: Compatible with popular ReLU-sparse models.
  • x86-64 CPUs with AVX2 instructions, with or without NVIDIA GPUs, under Linux.

Install

A quick way to get started (always check the official docs for the latest):

pip install -r requirements.txt # install Python helpers' dependencies

How it fits a local-AI stack

PowerInfer runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

Sources

Stats from GitHub, 2026-06-08.

Frequently asked

Quick answers to common questions

What is PowerInfer?

PowerInfer is a inference-server tool for local AI workloads. High-speed Large Language Model Serving for Local Deployment

Is PowerInfer free and open source?

Yes, PowerInfer has 9,540 GitHub stars and is licensed under MIT. You can self-host it for free on windows.

What platforms does PowerInfer support?

PowerInfer runs on windows.

What hardware do I need for PowerInfer?

The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. PowerInfer has 9,540 GitHub stars and an active community.

Does PowerInfer support GPU acceleration?

PowerInfer supports GPU acceleration via CUDA, Metal, or Vulkan depending on your platform. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.

What are the best alternatives to PowerInfer?

Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.

How much does PowerInfer cost?

PowerInfer is free-open-source. It is completely free and open source to self-host.

Pairs well with

Complementary tools, models, and hardware

Comments coming soon

Configure NEXT_PUBLIC_GISCUS_REPO_ID and NEXT_PUBLIC_GISCUS_CATEGORY_ID at giscus.app to enable.