Transcription
You've got a large language model running. With one user, time to first token is super quick. And at 10 users, we're starting to see latency climb. But then at a hundred, you're watching GPU memory spike and throughput tank. And every cycle is wasted money.
Your model is most likely not the culprit here, but rather how your memory is being used during inference. Specifically how the model stores and retrieves the context it's building token by token. Today, we're going to unpack two mechanisms for reducing wait time and cost associated with inferencing models. They are KV cache. And paged attention. They both come out of the open source inference engine, VLLM. And they've helped us make strides in what's possible for LLMs that we are inferencing at scale.
Now, by the end of this video, you'll know exactly how these two techniques work, why they work, and how you can configure them to get dramatically more throughput out of a GPU that you already have.
If you've used an LLM API, you've noticed this. You'll submit a long prompt and then wait for a bit. Then suddenly tokens will stream back fast. That lag before your first token is the model processing your prompt. This is the pre-fill phase. It's one of the two phases in LLM inferencing, and it's highly compute bound. This is where the model has to run your input through every transformer layer to build a mathematical representation of everything you said before it can produce a single output token.
Now, imagine that you're doing this for those 100 simultaneous users. And every request has its own growing context. And every step of token generation, the system has to reach back into GPU memory and retrieve that context. This is the decode phase. This is Memory Bound. And it's using KV cache. Now if memory is fragmented, if it's full, if it is being recomputed, it will become latency you can see.
Okay, now that we understand the two phases of LLM imprints, prefill, and decode, let's discuss KV cache and page detention. When a transformer generates a new token, every layer computes three things for each token it's attending to. A query, a value, and a key. Now, the query is the concurrent token asking, what's relevant? To me. And then the key and the value are answering that question. The problem is that in autoregressive generation, you're producing one token at a time, but you have to rerun attention over all previous tokens on every single step. Without a cache, generating a thousand-token response means that the thousandth token has to recompute the keys and values for all 999 tokens before it. The KV cache just stores those keys and value matrices from previous steps so they don't recompute what you already know. Each new token only needs to compute its own KVQ. Then it attends over the cached KV history. KV cache is a memory for compute trade-off, but it's proven to be worth it for long sequences.
Memory is the real bottleneck in LLM serving. So now let's look at how this naive serving allocates GPU memory. So a 13 billion parameter model, something like the Llama 13B, takes about 26 gigabytes of GPU memory just for its weights on an A140 gigabyte card. That's already 65% of our available memory. Just for the weights. That's VRAM before the user hits a single endpoint. Now, the remaining 35% has to support the KV cache for every active request. And here's where this traditional system starts to fall apart. They pre-allocate a fixed contiguous block of memory for each request based on the maximum possible output length. So if your max context is 2048 tokens, but the average user sends in 200 tokens and gets 300 back, that's 1,500 tokens of reserved memory just sitting empty per request. So research shows that these traditional systems are wasting about 60 to 80% of this 35% used for KV cache memory, leaving only a small bit actually usable.
While page detention treats GPU memory like OS treats RAM. Now, the KV cache is powerful, but it creates a new problem. How do you store it efficiently for many concurrent requests of wildly different lengths? Traditional systems store each request KV cache as a giant contiguous block, like reserving an entire hotel floor for a single guest. So if your max sequence length is 2,048 tokens, that's how much memory you have on reserve. Even if the user only generates 200 tokens, the rest is wasted and unavailable for anyone else. And this is called internal fragmentation. It can be the culprit for KV cache memory waste along with external fragmentation, where a request of varying lengths leave large gaps between allocations. So if a new request needs 500 tokens, you may very well have enough memory, but no continuous region big enough for this to work. Also, look out for redundant duplication where the system prompt is stored separately for every concurrent request.
Now, page attention eliminates each one of these issues and applies the exact same insight that operating systems use for RAM, virtual memory paging. And instead of one contiguous block, it breaks KV cache into small fixed page sizes by default, 16 tokens each. Now these pages can live anywhere in GPU memory, non-contiguous, allocated on demand. So a lightweight block table maps logical page addresses, and this is what the model sees, to physical page addresses where they actually are in VRAM so that your GPU memory allocation looks more like this. Where you have 65% still allocated to these weights, but then the rest of this 35%.
Finally, here are three things you can tune on your deployment to get the most out of your GPU. First, tune GPU memory utilization. This controls what fraction of remaining VRAM goes to KV cache. The default is 0.9. Push it to 0.95 on stable workloads to pack more concurrent requests and pull it to .8 if you're seeing OOM errors under load burst. And you can benchmark your specific model before you commit here. If you need something to do that, you can check out guide LLM. It's an open source part of the LLM project.
Second, enable prefix caching. Page detention hashes each KV block by its token sequence. Requests sharing a system prompt point to the same physical memory. VLLM computes and stores it once. In RAG pipelines, multi-turn chat and coding agents that are gonna hit rates of 75 to 95% are common, with time to first token dropping dramatically because shared prefill is skipped entirely.
Third, enable chunked prefill. For throughput heavy workloads, by default VLLM runs prefill to completion before resuming decode, which causes streamed tokens to stutter when long prompts arrive. Chunked prefilled batches inflate decode request first, then fills remaining compute budget with prefilled chunks. Production deployments have seen 50% throughput improvement. And you can also set max-num batch tokens to greater than 2048 alongside it.
Finally, a bonus feature for latency-sensitive workloads is enabling a speculative decoding model. Speculative-model. Now, during decode, your GPU has spare compute idle between memory reads. A small draft model. Proposes a series of output tokens. And then a larger model verifies these in one forward pass. And if they're good to go, they get sent forward. And if not, the wrong ones get corrected. Output quality here is mathematically identical to running the large model alone. And at very high concurrency, the gains shrink since the batch is already keeping the GPU busy. So reach for this when interactive latency matters more than raw throughput. The LLM also ships a zero cost Ingram speculator via dash dash speculative dash model Ingram. And that's for structured or repetitive outputs.
I hope this was helpful, and I'd love to know what the biggest memory-related problem you've hit in production AI development. Feel free to drop it in the comments and like and subscribe. Thanks.