Running & Serving Open Models Efficiently

The appeal of open weights is that the model can live on hardware you control. But "it runs on my machine" and "it serves ten thousand requests a second" are separated by a stack of engineering decisions — how the weights fit in memory, where the decode loop spends its time, how requests are batched, and where the trust boundary really sits. Get these right and an open model becomes cheaper, faster, and more private than a hosted call. Get them wrong and you inherit a slow, expensive box that still leaks data through the edges. This article walks the path from a single local process to a production serving tier, and shows the levers that actually move cost and latency.

Local is a boundary, not just a location

Running a model locally means three things happen on hardware you control: the weights are stored there, the runtime that uses them executes there, and the answer is produced there. The prompt need never cross the network. That is a real privacy and control gain — but it is a property of the boundary, not merely of the location. A local model with a plugin that calls a search API, a telemetry hook, or a file-sync client is no longer local in the sense that matters.

The trade is control for responsibility. When the computation is yours, so is the patching, the capacity planning, the security of the runtime, and the integrity of the files you loaded. The first discipline of running open models is therefore to draw the boundary explicitly and know every place data can cross it.

The decode loop and where time goes

A language model does not emit an answer in one shot. It runs two phases. Prefill reads your whole prompt at once and builds the internal state; it is compute-heavy but parallel. Decode then generates one token at a time, feeding each token back in to predict the next, until a stopping rule fires. Decode is the phase you feel: latency is roughly the number of output tokens divided by the tokens-per-second the hardware sustains.

The reason decode is memory-bound rather than compute-bound is the key-value cache (KV cache): to avoid recomputing attention over the whole sequence each step, the runtime stores the keys and values for every prior token. That cache grows with context length and with every concurrent request, and it competes with the weights for the same limited memory. Most serving cost and most latency cliffs trace back to this one structure.

Fitting the model in memory: precision

Whether a model runs at all is usually a memory question, and the biggest lever is numerical precision. Weights stored at 16-bit floating point (fp16) take roughly two bytes per parameter, so a 7-billion-parameter model needs about 14 GB just for weights — before any KV cache. Quantization stores those weights in fewer bits: 8-bit (int8) roughly halves the footprint, and 4-bit (int4) roughly quarters it, at a modest and often acceptable loss in quality.

Quantization is what lets a model that would not fit on a consumer GPU suddenly run on one. The engineering judgment is how far to push it: 4-bit is transformative for fitting a model on small hardware, but for quality-sensitive work you validate the quantized model on your own evaluations rather than assuming the loss is free. Precision is not a detail — it often decides which hardware class you need at all.

From running to serving

A single local process answers one request at a time. Serving many users needs a different shape: an inference server that keeps the weights resident in GPU memory and multiplexes many requests across them. Purpose-built servers — such as vLLM or Text Generation Inference (TGI) — add the machinery that makes this efficient: a paged KV cache that packs many sequences into memory without waste, and continuous batching that adds and removes requests from the running batch every step instead of waiting for a fixed batch to fill.

In front of the server sits the usual production apparatus: a gateway that authenticates and rate-limits, a load balancer across replicas, and streaming back to the client so tokens appear as they are generated. The trust boundary now wraps the whole tier, and everything inside it — the server, the cache, the GPUs — is yours to secure and scale.

Throughput versus latency

Serving is a negotiation between two goals that pull apart. Batching more requests together raises throughput — total tokens per second across all users — because the GPU does more useful work per pass. But a larger batch can raise the latency any single user feels, and a longer context per request consumes more KV cache, reducing how many requests fit at once. Continuous batching softens the trade by keeping the batch full without making early requests wait, but it does not abolish it.

The practical consequence is that you tune for a target, not for a universal best. An interactive assistant optimizes for low first-token latency and steady streaming; a bulk document-processing job optimizes for throughput and tolerates higher per-request latency. Knowing which you are building tells you how to set batch size, context limits, and replica count.

Choosing how to serve

The decision of how to run an open model follows a short branch. If the workload is latency-critical but low-volume — a developer tool, an offline task, a privacy-sensitive desktop use — a local runtime on capable hardware is often the simplest and most private answer. If volume is high, a batched inference server behind a gateway earns its complexity by driving down cost per token. And if neither the control nor the economics favor self-hosting, a hosted API remains the pragmatic choice, with the model still open enough to move later.

None of these is permanent. A common path is to prototype locally, prove value, then graduate to a served tier as volume grows — the open weights make that migration a deployment change, not a rewrite.

What to weigh

For a practitioner, running an open model well is mostly memory arithmetic and boundary hygiene: size the weights and KV cache against real hardware, pick a precision you have validated, choose a runtime or server that matches your concurrency, and account for every edge where data can leave. The decode loop and the KV cache are where your cost and latency actually live, so measure tokens per second and memory headroom under realistic load, not on a single-prompt demo.

For a leader, the frame is total cost of ownership against control. Self-hosting trades a hosted bill for capital, operations, and security responsibility — worth it at scale or where data cannot leave, wasteful at low volume. The durable advantage of open weights is optionality: you can start hosted, move to a served tier, or run locally, and change your mind as economics and requirements evolve.

Glossary

Runtime

The program that loads weights, tokenizes input, runs the calculation, and produces output text.

Token

The small unit of text — a word, word-piece, or symbol — that a model predicts one at a time.

Prefill

The first generation phase that reads the whole prompt in parallel to build internal state.

Decode

The phase that generates output one token at a time, feeding each back to predict the next.

KV cache

Stored keys and values for prior tokens that avoid recomputation but consume memory that grows with context and concurrency.

Quantization

Storing weights in fewer bits (e.g. int8 or int4) to shrink memory at a modest quality cost.

Continuous batching

A serving technique that adds and removes requests from the running batch each step to keep the GPU busy.

Inference server

Software such as vLLM or TGI that keeps weights resident and multiplexes many requests efficiently.

Throughput

Total tokens generated per second across all concurrent requests.