flashinfer

flashinfer is a local inference server - FlashInfer: Kernel Library for LLM Serving.

Overview

FlashInfer is a library and kernel generator for inference that delivers state-of-the-art performance across diverse GPU architectures. It provides unified APIs for attention, GEMM, and MoE operations with multiple backend implementations including FlashAttention-2/3, cuDNN, CUTLASS, and TensorRT-LLM.

• State-of-the-art Performance: Optimized kernels for prefill, decode, and mixed batching scenarios - Multiple Backends: Automatically selects the best backend for your hardware and workload - Modern Architecture Support: Support for SM75 (Turing) and later (through Blackwell) - Low-Precision Compute: FP8 and FP4 quantization for attention, GEMM, and MoE operations - Production-Ready: CUDAGraph and torch.compile compatible for low-latency serving

• BF16 GEMM: BF16 matrix multiplication for SM10.0+ GPUs. - FP8 GEMM: Per-tensor and groupwise scaling - FP4 GEMM: NVFP4 and MXFP4 matrix multiplication for Blackwell GPUs - Grouped GEMM: Efficient batched matrix operations for LoRA and multi-expert routing

• RoPE: LLaMA-style rotary position embeddings (including LLaMA 3.1) - Normalization: RMSNorm, LayerNorm, Gemma-style fused operations - Activations: SiLU, GELU with fused gating

Notable updates: - [2025-10-08] Blackwell support added in v0.4.0 - [2025-03-10] Blog Post Sorting-Free GPU Kernels for LLM Sampling, which explains the design of sampling kernels in FlashInfer.

See documentation for comprehensive API reference and tutorials.

FlashInfer provides comprehensive API logging for debugging. Enable it using environment variables:

flashinfer is open-source, written primarily in Python, with 5,760 GitHub stars under the Apache 2.0 license. The latest release is v0.6.12 (2026-05-29).

Key capabilities

From the project's documentation:

• State-of-the-art Performance: Optimized kernels for prefill, decode, and mixed batching scenarios
• Multiple Backends: Automatically selects the best backend for your hardware and workload
• Modern Architecture Support: Support for SM75 (Turing) and later (through Blackwell)
• Low-Precision Compute: FP8 and FP4 quantization for attention, GEMM, and MoE operations
• Production-Ready: CUDAGraph and torch.compile compatible for low-latency serving
• Paged and Ragged KV-Cache: Efficient memory management for dynamic batch serving

Install

A quick way to get started (always check the official docs for the latest):

pip install flashinfer-python

How it fits a local-AI stack

flashinfer runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

• ExLlamaV2
• llama.cpp
• LocalAI
• Ollama

Sources

• Source code & docs: flashinfer-ai/flashinfer
• Official website: https://flashinfer.ai

Stats from GitHub, 2026-06-08.