ghost-infer
Fast local LLM inference for the GPU you already have: one Rust binary, GGUF models, an OpenAI-compatible server and a chat page with a ghost.
What it does
Core capabilities at a glance
- Cuda
- Desktop APP
- Gguf
- LLM Inference
- Local LLM
- Mixture OF Experts
- Openai Compatible API
- Windows
Deep dive
The full breakdown - performance, comparisons, and setup
ghost-inferLink to this section
ghost-infer is a local inference server - Fast local LLM inference for the GPU you already have: one Rust binary, GGUF models, an OpenAI-compatible server and a chat page with a ghost.
OverviewLink to this section
Fast local LLM inference for the GPU you already have. One small Rust program that runs GGUF models on an NVIDIA card, serves an OpenAI-compatible API, and comes with a desktop app where the model works on your files.
Most inference engines are built for data centres: many users, big GPUs, batching. ghost-infer is built for the other case: one person, one consumer GPU, one model at a time, and it is tuned for exactly that.
ghost-infer 0.4.0, llama.cpp b11747 and Ollama 0.40.13, all measured 2026-11-14 in one run ('scripts/bench-all.ps1'). Qwen3 8B: GuideLLM, a ~264-token chat prompt and 128 tokens. The A3B rows: seven varied prompts (code, an explanation, a code review, a summary, a chat, a word problem, SQL) through each engine's API, the median of two rounds, llama.cpp at its best '--n-cpu-moe' setting (24); i1-Q3_K_M files. Rows marked *: measured 2026-11-10 with 16 GB of RAM and ghost-infer 0.3.0, tokens/s writing code / writing a long answer, the second (warm) of two runs, llama.cpp with its automatic GPU split. Method and more results in Benchmarks.
ghost-infer is open-source, written primarily in Rust, with 249 GitHub stars under the Apache 2.0 license. The latest release is v0.4.0 (2026-11-13).
Key capabilitiesLink to this section
From the project's documentation:
- An agent on your files. The model reads and edits your files and runs commands, asking first,
- Voice calls. Talk with Maya, a companion who answers aloud and remembers your calls, with her own
- English or Chinese. The desktop app's own words come in either language: Settings, General, Language,
- About twice as fast on models bigger than the GPU. Each layer runs as one CUDA graph, the GPU picks its own
- Faster sampling. Picking a token with temperature, top-k or a repetition penalty no longer scans the whole
- Ternary Bonsai 27B on the GPU (Q2_0 and PTQ1_0), about 40 tokens/s on an 8 GB card.
How it fits a local-AI stackLink to this section
ghost-infer runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:
SourcesLink to this section
- Source code & docs: affifuddin-hazam/ghost-infer
Stats from GitHub, 2026-11-14.
Frequently asked
Quick answers to common questions
What is ghost-infer?
ghost-infer is a inference-server tool for local AI workloads. Fast local LLM inference for the GPU you already have: one Rust binary, GGUF models, an OpenAI-compatible server and a chat page with a ghost.
Is ghost-infer free and open source?
Yes, ghost-infer has 249 GitHub stars and is licensed under Apache 2.0. You can self-host it for free on windows.
What platforms does ghost-infer support?
ghost-infer runs on windows.
What hardware do I need for ghost-infer?
The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. ghost-infer has 249 GitHub stars and an active community.
Does ghost-infer support GPU acceleration?
ghost-infer supports GPU acceleration via CUDA, Metal, or Vulkan depending on your platform. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.
What are the best alternatives to ghost-infer?
Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.
How much does ghost-infer cost?
ghost-infer is free-open-source. It is completely free and open source to self-host.
Pairs well with
Complementary tools, models, and hardware