Running models locally

Running a model locally means downloading an open-weight model’s parameters and running them on your own hardware instead of calling a hosted API. llama.cpp, one runtime built for this, describes itself in four words (“LLM inference in C/C++”) and states its goal as inference “with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud”[2]. Why: privacy (nothing leaves the machine), cost at real volume (no per-token bill), working offline, and control over exactly which model version runs. What it costs instead: hardware you buy and keep running, quality capped by what fits on that hardware, and the upkeep of a system component you now run yourself rather than a vendor.

This topic is not a level on the ladder; it applies at every level. What changes by level is how much context and how many concurrent requests the hardware has to hold, which is this page’s own “At each level” section below.

This page is sourced, not measured: the sizes and requirements below come from the model makers’ and runners’ own pages, and nothing here has been timed on hardware and scored.

def weights_bytes(params: int, bits_per_weight: float) -> int:
    """Parameter count times bits per weight, converted to bytes. `bits_per_weight` is the
    caller's own average figure: 16 for fp16, roughly 4 for a 4-bit quantization such as Q4_K_M.
    A real quantized file usually averages a little above the bit width in its name, because not
    every tensor is quantized the same way (llama.cpp's quantize tool can leave the output tensor
    unquantized, for instance). This function models none of that; it takes the average given."""
    return round(params * bits_per_weight / 8)

def kv_cache_bytes(
    *, context_length: int, num_layers: int, num_kv_heads: int, head_dim: int, bytes_per_value: int = 2, num_sequences: int = 1
) -> int:
    """The key/value cache: two tensors (key and value) per layer, each sized
    context_length x num_kv_heads x head_dim at bytes_per_value bytes, times how many sequences
    are served at once. `num_kv_heads` is the key/value head count from the model's own config,
    which grouped-query attention makes smaller than the attention head count."""
    per_sequence = 2 * num_layers * context_length * num_kv_heads * head_dim * bytes_per_value
    return per_sequence * num_sequences

python -m examples.local_inference --demo

Speculative decoding

Quantization makes a model fit. Speculative decoding is the other lever a local server has, and it changes what the same model costs per token rather than what it weighs. A small draft model proposes the next few tokens, the large one checks them in a single pass, and a sampler keeps the ones the large model would have produced itself and throws the rest away. vLLM’s own guide states the regime it is for: to “reduce inter-token latency under medium-to-low QPS (queries per second), memory-bound workloads”[5]. Both runtimes this page cites document it. llama.cpp’s server lists “Speculative decoding” among its features, with --spec-draft-model documented as the “draft model for speculative decoding (default: unused)” and --spec-draft-n-max as the “number of tokens to draft for speculative decoding (default: 3)”[6].

No decision moves, at any level, which is why this is a topic and never a rung. The output is meant to be the one the large model would have given on its own. What it costs is memory, in the terms this page’s own estimator already uses: a second set of weights and a second key/value cache, which llama.cpp exposes as their own settings for the draft model’s cache type and its layers in VRAM[6]. Size for both, or the drafting that was supposed to buy speed takes the memory the context window needed.

You do not need it if your bottleneck is fitting the model at all, or if the hardware is already saturated by concurrent requests rather than waiting between tokens.

The failure mode to know is that “identical output” is a guarantee with edges. vLLM states that “Speculative decoding sampling is theoretically lossless up to the precision limits of hardware numerics”, and, in the same section, that “variations in generated outputs with and without speculative decoding can occur due to following factors”, naming floating-point precision and batch size[5]. So a golden set that passed before it was turned on is worth re-running after, rather than assumed.

Failure modes

At each level

• Conventional software: nothing to run locally: this topic’s own concerns start at level 1.
• Direct prompting: the smallest case (one short prompt, one short answer), the least memory this page’s own estimator will ever report for a given model.
• Added context: a retrieved passage set or a large pasted document fills the context window before the model answers at all, so context length, not just model size, starts to drive the memory number. This is the same window RAG’s own prompt assembly fills.
• Tool use: tool definitions and their results are extra tokens in the same context window, so a tool-heavy function calling run costs more context, and so more KV cache, than the same question answered with no tools at all.
• Agent loops: a long-running loop’s transcript keeps growing across turns, so the KV cache a single agent run needs grows with it: a context budget sized for one exchange is not sized for a whole run.
• Teams of Agents: several agents worked at once on one machine are exactly the “several sequences” case this page’s own num_sequences parameter counts: a lead and its workers running locally multiplies the KV cache cost by however many are active together, not just by one.
• Always-on agents: load is sustained rather than one burst per question, so the number to size for is how many sequences are live at the same time on an ordinary day, which is a different question from how large the largest single request is.

Practices

• Size from the model’s own config, not from its name: layer count, key/value head count and head dimension are what the cache formula needs, and grouped-query attention makes the third of those smaller than the head count people usually quote.
• Estimate at the longest context and the highest concurrency the system is meant to reach, not at a typical one. Both terms are linear, so the worst case is easy to compute and easy to skip.
• Treat any estimate, this one included, as a floor. Measure what the runtime actually reports and keep the gap; that gap is the activation memory and overhead nobody’s formula counts.
• Decide quantization with an eval on your own task, and record the exact quantization you ran, not “4-bit”.
• Compare against what the hosted alternative costs over the same period before buying anything: that arithmetic belongs to cost optimization, and hardware is a fixed cost that does not care whether you use it.

Run it

Try it

1. Use it

   If you have a local model tool installed (or are willing to install one), check what context size or "memory" setting it exposes, and what it says that costs in hardware. If it doesn't say, that silence is itself worth noting.

2. Build it

   Run python -m examples.local_inference --demo from the repo root and compare the KV cache column across the four rows. Then add a fifth shape of your own with a 128K context and one user. Going from 32K to 128K is four times the context: predict the KV cache figure before you run it, then check whether the printed number is four times the 32K row.

3. Either lane

   Pick a model you know the approximate parameter count of. Using this page's estimator (or your own arithmetic from weights_bytes' formula), compute its memory footprint at full precision and at roughly 4-bit quantization, and compare the difference against what hardware you actually have.