inside-llm
See inside any GGUF: where two quantizations part ways, token by token, and what each sampler cuts. Live in your browser, powered by llama.cpp (CUDA or Vulkan).
What it does
Core capabilities at a glance
- CPP
- Cuda
- Gguf
- Interpretability
- Llama CPP
- Local LLM
- Logit Lens
- Quantization
Deep dive
The full breakdown - performance, comparisons, and setup
inside-llmLink to this section
inside-llm is a local inference server - See inside any GGUF: where two quantizations part ways, token by token, and what each sampler cuts. Live in your browser, powered by llama.cpp (CUDA or Vulkan).
OverviewLink to this section
See inside any GGUF: where two quantizations part ways, token by token, and what each sampler cuts. Live, in your browser.
'inside-llm' with Qwen3.5-4B at two quantizations, Q2_K and Q8_0: the token where their texts split, the candidates behind each pick, the sampler chain, branches and the relief of probabilities. Every number comes from the real engine running the two models.
'inside-llm' loads one or two GGUF models with llama.cpp and shows in the browser what the model does while it generates: the probability of each token, the alternatives it discarded, what each sampler removes and, with two models (two quantizations, or a base model and its fine-tune), the divergence at every position and the exact token where their texts split. It is a single native binary that serves its own web page. Everything runs on your machine: no accounts, no API keys, no telemetry. A session can be recorded and replayed later in a static page, with no engine and no GPU.
Try the live demo: recorded sessions replayed in your browser, nothing to install.
inside-llm is open-source, written primarily in TypeScript, with 110 GitHub stars under the MIT license. The latest release is v1.0.0 (2026-11-15).
Key capabilitiesLink to this section
From the project's documentation:
- Metrics: confidence, surprise, perplexity, margin, speed and latency of A and B, as sparklines along the text.
- Model A and model B must share the tokenizer.
- Both models must fit in GPU memory at once; the picker estimates it before loading.
- One GPU at a time. Integrated GPUs are never used; without a dedicated GPU it runs on the CPU, much slower.
- Text only: multimodal models load, but only their text side is shown.
- The engine is meant for your own machine: it listens on localhost and has no authentication.
How it fits a local-AI stackLink to this section
inside-llm runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:
SourcesLink to this section
- Source code & docs: manuelgscom/inside-llm
- Official website: https://manuelgs.com/en/inside-llm/
Stats from GitHub, 2026-11-19.
Frequently asked
Quick answers to common questions
What is inside-llm?
inside-llm is a inference-server tool for local AI workloads. See inside any GGUF: where two quantizations part ways, token by token, and what each sampler cuts. Live in your browser, powered by llama.cpp (CUDA or Vulkan).
Is inside-llm free and open source?
Yes, inside-llm has 110 GitHub stars and is licensed under MIT. You can self-host it for free on linux, windows, web.
What platforms does inside-llm support?
inside-llm runs on linux, windows, web.
What hardware do I need for inside-llm?
The hardware requirements depend on which models you run. Check our hardware directory for compatible GPUs and systems. inside-llm has 110 GitHub stars and an active community.
Does inside-llm support GPU acceleration?
inside-llm supports GPU acceleration via CUDA, Metal, or Vulkan depending on your platform. For the best performance, pair it with an NVIDIA RTX 4090 or 5090.
What are the best alternatives to inside-llm?
Popular alternatives include other inference-server tools in our directory. Browse our full collection at /tool for comparisons, community reviews, and benchmark data to find the right fit for your workflow.
How much does inside-llm cost?
inside-llm is free-open-source. It is completely free and open source to self-host.
Pairs well with
Complementary tools, models, and hardware