RustingStrata
Pure-Rust LLM inference engine for GGUF (Qwen, DeepSeek, GLM, Hy4, Flint). Runs models bigger than your VRAM on CUDA and Vulkan, with MTP drafts and an OpenAI/…
What it does
Core capabilities at a glance
- Anthropic API
- Cuda
- Gguf
- Inference
- Llama CPP
- Local LLM
- Mixture OF Experts
- MOE
Deep dive
The full breakdown - performance, comparisons, and setup
RustingStrataLink to this section
RustingStrata is a local inference server - Pure-Rust LLM inference engine for GGUF (Qwen, DeepSeek, GLM, Hy4, Flint). Runs models bigger than your VRAM on CUDA and Vulkan, with MTP drafts and an OpenAI/Anthropic server. 12 GB RTX 3060 vs llama.cpp: Qwen3.6-35B-A3B UD-Q5_K_M 80.5 vs 64.4 tok/s, Flint UD-Q5_K_XL 64.6 vs 48.6. With MTP: 108 tok/s (UD-Q3_K_M), up to 160 (UD-Q2_K_XL).
OverviewLink to this section
A pure-Rust inference engine for Qwen GGUF models: Qwen3.5/3.6 ('qwen35moe', 'qwen35'), Qwen3-Next ('qwen3next'), Qwen3.8-Flash-Next ('qwen4exp'), Qwen3 ('qwen3', 'qwen3moe'), DeepSeek-V4.1-Flash ('deepseek41'), GLM-5.3-Flash ('glm5-next'), Hy4-flash ('hy_v4_flash') and Recursive Flint ('flint'). CUDA kernels are compiled at run time with NVRTC through cudarc, and the same kernels also run on Vulkan ('--backend vulkan'): no C++ build step, no ggml. Experts that do not fit in VRAM run on the CPU at the same time as the GPU work, and the expert placement adapts to the routing it sees.
Version 1.2.6. Docs: usage and server API, hosting for other people, supported GPUs, AMD notes, LoRA, all benchmarks, benchmark history.
Measured at commit 5a27ed1 (1.2.5) with 'rustingstrata serve --gpu --ctx 8192 --mtp 0' and 'llama-server': the same 4241-token prompt file and the same script for both, greedy, thinking off, 128 tokens, no drafts in either engine. Two loads per engine in alternating order; in each load a warm-up and the first 4K request are thrown away. Decode is the mean of runs 4-8 of each load (10 runs, range in brackets). Both engines climb for the first runs after a load; over all 16 runs it is 78.8 (63.6-82.0) against 60.3 (31.3-66.0). Prefill is 4 new prompts per load (8 runs).
RustingStrata is open-source, written primarily in Rust, with 315 GitHub stars under the Other license. The latest release is v1.2.6 (2026-11-19).
Key capabilitiesLink to this section
From the project's documentation:
- an NVIDIA driver with the CUDA NVRTC library (loaded at run time). Volta (sm_70), Turing (sm_75) and Ampere or newer
- a Vulkan 1.3 driver with shaderFloat64, int8/16/64 and subgroup size 32 or a fixed 64 (tested on RTX 3060, Radeon VII
How it fits a local-AI stackLink to this section
RustingStrata runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:
SourcesLink to this section
- Source code & docs: RustingStudio/RustingStrata
Stats from GitHub, 2026-11-21.
Frequently asked
Quick answers to common questions
What is RustingStrata?
RustingStrata is a inference-server tool for local AI workloads. Pure-Rust LLM inference engine for GGUF (Qwen, DeepSeek, GLM, Hy4, Flint). Runs models bigger than your VRAM on CUDA and Vulkan, with MTP drafts and an OpenAI/…
Is RustingStrata free and open source?
Yes, RustingStrata has 315 GitHub stars and is licensed under Other. You can self-host it for free on .
What hardware do I need for RustingStrata?
The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. RustingStrata has 315 GitHub stars and an active community.
Does RustingStrata support GPU acceleration?
RustingStrata's GPU support depends on your specific setup. Check the documentation for details. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.
What are the best alternatives to RustingStrata?
Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.
How much does RustingStrata cost?
RustingStrata is free-open-source. It is completely free and open source to self-host.
Pairs well with
Complementary tools, models, and hardware