LLM VRAM: Weights, KV Cache and Context

Why a local model can load but fail on a longer prompt. Separate weights, KV cache and runtime memory with worked arithmetic and a diagnostic worksheet.

By · Updated
The short answer

A successful model load does not prove a longer conversation fits. Budget resident weights, KV cache and runtime working memory separately. Context length, cache format and concurrent sequences change the result; validate the actual runner and workload.

Why a model loads but a long prompt failsLink to this section

The download size describes a model artifact. It does not describe the peak memory needed to run your conversation. A model can load successfully and still exhaust device memory when context or concurrency increases.

For GPU-resident inference, keep three entries in your budget:

Resident weights + resident KV cache + runtime working memory.

Leave room for the desktop and other GPU processes too. CPU offload changes where allocations live. Unified memory shares capacity with the operating system and applications; it is not a second dedicated VRAM pool you can add to a GPU specification.

This guide explains the accounting with a hypothetical decoder. It does not report a hardware test or recommend a GPU purchase. For a first setup, see the local AI getting-started guide. For candidate selection, browse the model directory.

Weight size is only the first allocationLink to this section

An ideal four-bit representation of eight billion parameters occupies four billion bytes before scales, metadata, padding and tensors stored at higher precision. That calculation is not a prediction of a real GGUF download size. Inspect the exact artifact rather than multiplying its marketing parameter count by one universal coefficient.

Weight quantization does not automatically change cache precision. A four-bit weight file can still use a two-byte-per-element cache. Check the runner's supported cache formats and effective configuration separately.

Runtime working memory includes intermediate tensors, execution buffers and temporary allocations. Peak usage depends on the backend and workload. There is no fixed allowance that proves every model fits. The VRAM calculator is a starting estimate, with important limits described below.

What changes as context growsLink to this section

Autoregressive attention can retain keys and values from earlier tokens. A dynamic full-attention cache grows with retained history. A static cache can reserve its configured capacity upfront. Sliding-window layers may retain only a bounded history, so their cache does not grow indefinitely with the whole conversation. These policies are documented in Hugging Face's cache strategies.

That means a short first message does not always imply a small reservation. Inspect what the backend allocates, not only what the chat box currently shows. Offloaded caches also change placement; a smaller GPU allocation is not proof of a smaller total memory footprint or faster inference.

A worked KV-cache calculationLink to this section

For a simplified full-attention decoder with identical cache geometry in every layer:

cache_bytes = 2 × layers × KV_heads × head_dimension
              × retained_tokens × bytes_per_element × sequences

The factor two counts keys and values. Use the number of KV heads, which can differ from the number of query heads. The sequences term assumes independent sequences of equal length, without shared-prefix savings. Padding, metadata and allocation policies can add memory beyond this payload.

Take a hypothetical decoder with 32 layers, eight KV heads per layer, head dimension 128 and two bytes per cache element. Assume full attention in all layers, one sequence and no compression. Its per-token payload is:

2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB
Retained tokensOne sequence: calculated cache payloadFour independent sequences
4,0960.5 GiB2 GiB
8,1921 GiB4 GiB
32,7684 GiB16 GiB

These figures exclude weights, working memory and other processes. GiB is 1,073,741,824 bytes; GB is 1,000,000,000 bytes. No named model or GPU was measured for this table.

Four sequences do not necessarily mean four copies of the weights. Serving engines can share weights while keeping separate request state. Equally, the four-sequence column is not a universal estimate for any server: prefix sharing, unequal lengths and cache scheduling change allocation.

Do not transplant this coefficient to a model with a different layer layout, cache format, sliding-window policy or attention mechanism. The useful habit is to expose the assumptions before trusting the total.

Check three different context limitsLink to this section

  1. Model support: the context behavior supported by the model and its configuration.
  2. Runner allocation: the capacity and cache policy configured in the inference backend.
  3. Application budget: the tokens the application actually sends, retains and requests as output.

A model's advertised maximum is not evidence that your GPU can serve that many tokens. A document application may retrieve a few passages instead of sending the whole file. A chat template, system prompt and tool results can consume space before your visible message. Record the actual request, not just the document's page count.

For Ollama, ollama ps shows allocated context and CPU/GPU placement. Its context-length documentation describes those fields and confirms that increasing context increases memory requirements. Do not assume a default remains identical across versions or devices. Other runners, including LM Studio and vLLM, expose different controls; consult the documentation for your installed version.

A worksheet for an out-of-memory reportLink to this section

Download the VRAM and context worksheet (CSV). Open it in a spreadsheet and save a separate copy for each workload. Fill only the blank measured-input rows with your own observations, then change one variable and repeat the same input. The populated hypothetical rows reproduce the cache arithmetic above; they are not measurements or a model-fit prediction. Memory readings use GiB; illustrative payloads include both GiB and decimal GB, clearly separated.

Record these fields before changing several settings at once:

FieldWhat to record
ModelExact artifact, revision and weight quantization
RuntimeRunner version, GPU backend and offload settings
Token budgetConfigured context, actual prompt tokens and requested output tokens
CachePrecision, placement, attention policy and concurrent sequences
MemoryAvailable memory before load, after load and at workload peak
OutcomeLoad success, full-request success and whether the answer remains useful

Repeat the same input after reducing one variable, such as configured context or concurrency. Capture the runner's error and memory report. This is a proposed troubleshooting procedure; no such run was performed for this guide.

If reducing context fixes memory use, also verify that the task still receives the information it needs. Truncating an important passage can make a run finish while making its answer worse. Cache quantization and offload depend on backend support and can involve quality or performance tradeoffs. Successful loading alone is an incomplete acceptance test.

How to interpret our calculatorLink to this section

Every Local AI's current calculator uses approximate bytes per parameter, a fixed per-token cache coefficient and a fixed runtime allowance, then rounds upward. It does not derive each model's cache geometry or account for every cache dtype, attention layout and concurrency setting. Treat its result as a heuristic planning estimate, not an exact-fit guarantee.

Use it to narrow a candidate list, then inspect the real runner and validate the workload. Keep a measured memory report separate from a calculated estimate. Browse hardware specifications to identify the relevant memory pool; the benchmark index contains separate experiments with their own methods, not validation of this hypothetical example.

Before sharing that a model "fits," add three details: which context, which cache format and how many concurrent sequences?

Prepared with AI assistance. Documentation and calculator source inspected on 14 November 2026. All numeric examples above are derived arithmetic, not measured performance.

Frequently asked

Does a 4-bit model automatically use a 4-bit KV cache?

No. Weight precision and cache precision are separate settings. Check whether the inference backend supports the cache format you intend to use.

Does a model's maximum context length prove it fits my GPU?

No. Model support, runner allocation and application token budget are separate limits. Memory depends on the architecture, cache policy, workload and other allocations.

Are the numbers in this guide measured benchmarks?

No. The table is derived arithmetic for an explicitly hypothetical decoder. It excludes weights, runtime overhead and other processes, and makes no GPU compatibility or speed claim.