lucebox-hub

lucebox-hub is a local inference server - Fast LLM speculative inference server for consumer hardware.

Overview

Local LLM inference server built for speed. Custom kernels, speculative prefill & decoding. Each optimization in our engine is for specific model family and hardware target.

Each one is self-contained with setup instructions and benchmark notes.

All speedups measured vs vendored llama.cpp ('-fa 1', matching KV quant). Combined = geometric mean √(TTFT × decode) where both phases benched; otherwise the single-phase speedup. Drafters published on huggingface.co/Lucebox.

Reference target: RTX 3090 (Ampere sm_86) — all headline numbers. Other NVIDIA archs auto-detected by CMake / 'setup.py'; AMD HIP backend separate (Strix Halo section).

'server/' (DFlash) builds with CMake 3.18+ and '--recurse-submodules' for 'Luce-Org/llama.cpp@luce-dflash' — no PyTorch needed. 'optimizations/megakernel/' is the only component requiring PyTorch 2.0+ (CUDAExtension links against torch C++ libs). Power-tune: 'sudo nvidia-smi -pl 220' (3090 sweet spot, re-sweep for other cards).

'harness/' contains RTX 3090 client launchers and regression tests for Lucebox server compatibility. Run Lucebox inside Claude Code, Codex, OpenCode, Hermes, Pi, OpenClaw, or Open WebUI, or check if a server change still works with those clients.

All launchers spawn the native C++ HTTP server ('dflash_server'). Override defaults via env vars:

For no-draft targets such as Gemma, set only 'DFLASH_TARGET' or pass 'DRAFT=none'; the harness will not attach the default Qwen draft to a custom target.

Launcher scripts install missing real-client CLIs automatically under '.harness-work/'. To preinstall them yourself:

lucebox-hub is open-source, written primarily in C++, with 2,341 GitHub stars under the Apache 2.0 license. It was last updated on 2026-06-08.

How it fits a local-AI stack

lucebox-hub runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

• ExLlamaV2
• llama.cpp
• LocalAI
• Ollama

Sources

• Source code & docs: Luce-Org/lucebox-hub
• Official website: https://www.lucebox.com

Stats from GitHub, 2026-06-08.