kserve

kserve is a local inference server - Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes.

Overview

KServe is a standardized distributed generative and predictive AI inference platform for scalable, multi-framework deployment on Kubernetes.

KServe is being used by many organizations and is a Cloud Native Computing Foundation (CNCF) incubating project.

Single platform that unifies Generative and Predictive AI inference on Kubernetes. Simple enough for quick deployments, yet powerful enough to handle enterprise-scale AI workloads with advanced features.

To learn more about KServe, how to use various supported features, and how to participate in the KServe community, please follow the KServe website documentation. Additionally, we have compiled a list of presentations and demos to dive through various details.

• Standard Kubernetes Installation: Compared to Serverless Installation, this is a more lightweight installation. However, this option does not support canary deployment and request based autoscaling with scale-to-zero. - Knative Installation: KServe by default installs Knative for serverless deployment for InferenceService. - ModelMesh Installation: You can optionally install ModelMesh to enable high-scale, high-density and frequently-changing model serving use cases. - Quick Installation: Install KServe on your local machine.

KServe is an important addon component of Kubeflow, please learn more from the Kubeflow KServe documentation. Check out the following guides for running on AWS or on OpenShift Container Platform.

kserve is open-source, written primarily in Go, with 5,552 GitHub stars under the Apache 2.0 license. The latest release is v0.18.0 (2026-04-29).

Key capabilities

From the project's documentation:

• 🧮 Optimized Backends: Support for vLLM and llm-d for optimized performance for serving LLMs
• 📌 Standardization: OpenAI-compatible inference protocol for seamless integration with LLMs
• 🚅 GPU Acceleration: High-performance serving with GPU support and optimized memory management for large models
• 📈 Autoscaling: Request-based autoscaling capabilities optimized for generative workload patterns
• 🔧 Hugging Face Ready: Native support for Hugging Face models with streamlined deployment workflows
• 🧮 Multi-Framework: Support for TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, and more

How it fits a local-AI stack

kserve runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

• ExLlamaV2
• llama.cpp
• LocalAI
• Ollama

Sources

• Source code & docs: kserve/kserve
• Official website: https://kserve.github.io/website/

Stats from GitHub, 2026-06-08.