What it does
Core capabilities at a glance
- Cuda
- Gguf
- Linux
- Llama CPP
- LLM Inference
- Local LLM
- Macos
- Metal
Deep dive
The full breakdown - performance, comparisons, and setup
inferssdLink to this section
inferssd is a local inference server - Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
OverviewLink to this section
Download app · Website · Blog · Discord · Slack · Apache-2.0
Run models larger than your RAM or VRAM. InferSSD builds on llama.cpp to run large mixture-of-experts (MoE) models from SSD, using standard GGUF files, with browser chat and an OpenAI-compatible API. It runs DeepSeek-V4.1-Flash-1017 and Qwen3.8 Flash Next in many quantizations, as well as Muse-Spark-Open, Qwen3.6-35B-A3B and Gemma4 E4B.
These are the tested GGUF files, used as their publishers released them. Expand a model for its files.
mrfakename's Q8_0 keeps the official bit widths: routed experts stay native 4-bit MXFP4, and FP8 tensors become 8-bit Q8_0. See weight-format details.
4K prompt tier · 256 output tokens · decode measured over tokens 129–256. All rates are in token/s; higher is faster. The multiple next to each InferSSD rate is its speedup over llama.cpp on the same hardware. On every machine, llama.cpp and InferSSD run DeepSeek's mrfakename Q8_0 and Muse's Unsloth UD-Q4_K_XL; on the MacBook Pro they run them with Metal, and Colibri runs the official Safetensors on CPU. See measurement details.
Unified memory is shared by the CPU and GPU. SSD capacities are nominal. See hardware details for SSD models, CPU core layouts, and configuration notes.
Hardware Prefill (token/s) Decode (token/s) llama.cpp Colibri InferSSD llama.cpp Colibri InferSSD
Hardware Prefill (token/s) Decode (token/s) llama.cpp Colibri InferSSD llama.cpp Colibri InferSSD
inferssd is open-source, written primarily in C++, with 1,276 GitHub stars under the Apache 2.0 license. The latest release is v0.1.5 (2026-11-18).
Key capabilitiesLink to this section
From the project's documentation:
- Versions and engine resources
InstallLink to this section
A quick way to get started (always check the official docs for the latest):
curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | shHow it fits a local-AI stackLink to this section
inferssd runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:
SourcesLink to this section
- Source code & docs: wici-ai/inferssd
- Official website: https://wici.ai/wici-infer
Stats from GitHub, 2026-11-19.
Frequently asked
Quick answers to common questions
What is inferssd?
inferssd is a inference-server tool for local AI workloads. Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
Is inferssd free and open source?
Yes, inferssd has 1,276 GitHub stars and is licensed under Apache 2.0. You can self-host it for free on macos, linux, windows, web.
What platforms does inferssd support?
inferssd runs on macos, linux, windows, web.
What hardware do I need for inferssd?
The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. inferssd has 1,276 GitHub stars and an active community.
Does inferssd support GPU acceleration?
inferssd supports GPU acceleration via CUDA, Metal, or Vulkan depending on your platform. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.
What are the best alternatives to inferssd?
Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.
How much does inferssd cost?
inferssd is free-open-source. It is completely free and open source to self-host.
Pairs well with
Complementary tools, models, and hardware