On-call Best Practices for SREs (Sponsored)
On-call shouldn’t feel like constant firefighting. This guide from Datadog breaks down how high-performing SRE teams reduce alert fatigue, streamline incident response, and design rotations that don’t burn engineers out.
You’ll learn how to:
Cut alert noise by tying signals to real user impact
Improve response with clear roles and smarter escalation paths
Turn incidents into feedback loops that improve system reliability
Why does sending a model a 100K-word prompt cost so much more than sending it a short one, even when the model and the hardware stay the same?
A key part of the answer lies in a block of working memory called the KV cache.
This memory is built up while the model generates a response. It is separate from the knowledge stored in the model’s weights, and it holds the key and value vectors computed for every token of the input. The cache grows with every token, and in a long context, it can take up significant space on the GPU.
For example, for a 70-billion-parameter model at a context of 128,000 tokens, it comes to roughly 40 gigabytes, a serious amount of GPU memory that grows with every user you add.
The above chart raises an obvious question, which is why a cache exists at all and why it grows the way it does. In this article, we will learn how LLMs use memory, how it gets expensive, and how to fix it.
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Recomputation
Let us start with the work a model does to produce one token.
To choose the next word, it runs an attention step, where the newest token compares itself against every token that came before it. This comparison uses two vectors for each earlier token, a key and a value, which are simply the numerical summaries the model computes for that token inside each layer. A model that rebuilt the key and value for every earlier token at every step would watch the work per token climb as the input grows. Such a repetition is pure waste, because those keys and values stay the same once a token has been processed.
The KV cache removes the waste by storing those key and value vectors the first time they are computed. On the next step, the model computes the key and value for only the new token and reads the rest straight from the cache.
Overall, this is a great solution. However, caching fixes the speed problem while creating a new one. The cache now has to be read on every step, and this turns out to be the real source of increasing cost.
One detail worth understanding here is that the cache holds vectors rather than the original text. It also explains out-of-memory errors that look puzzling when the model itself fits with room to spare.
Decoding
Generation of tokens runs in two phases, and they stress the hardware in different ways:
The first phase is prefill, where the model reads the entire input at once. It processes all of the input tokens in parallel and builds their key and value vectors into the cache in a single pass. Prefill keeps the GPU’s math units busy, so we call it compute-bound, meaning the limit is how fast the chip can do arithmetic.
The second phase is decoding, where the model produces the output one token at a time. Each new token runs an attention step against the whole cache, which means the model reads every stored key and value out of GPU memory before it can emit the next token. It repeats that read for every token it produces. The limit here is how fast the cache can move from memory into the compute units, so we call decoding memory-bound.
The expense of long-context generation comes less from holding the cache and more from sweeping through all of it on every single token.
A larger cache means more data crossing the memory bus per token, which shows up directly as slower and costlier generation. It also explains why a request can run slowly even when it fits in memory comfortably.
If the cost tracks how much of the cache we read each step, then the size of the cache is the next thing to understand clearly.
Scaling
The cache size is the product of a handful of numbers:
A factor of two covers the key and the value.
Layers count because each layer keeps its own cache.
Key-value heads set how many sets each layer stores.
Head dimension is the size of each of those vectors.
Bytes per number is the space one stored value takes.
Tokens are the context length, with one entry each.
Batch size is the number of requests served at once.
To summarize, the cache size equals 2 times layers times key-value heads times head dimension times bytes per number times tokens times batch size.
Two things that should be noted here are as follows:
The cache grows in a straight line with the token count, so doubling the context doubles the cache
It grows the same way with the batch size, so serving more users at once scales it just as fast.
As an approximate example, a Llama 3 70B model has 80 layers, 8 key-value heads, a head dimension of 128, and stores each number in 2 bytes. At a context of 128,000 tokens for a single request, those numbers multiply out to roughly 40 gigabytes, which is why a single long request can fill most of an 80-gigabyte card on its own.
Let us now look at the optimization techniques that help an LLM manage the memory aspects. Each of these techniques tries to push against a specific number.
Attention
The first two techniques change how attention itself is built, which means a model is committed to them during training. Both shrink the footprint of each token in the cache.
Grouped-query attention goes after the key-value head count. In a standard attention layer, every query head carries its own key and value head, so a model with 64 query heads stores 64 sets of keys and values. Grouped-query attention lets several query heads share one key-value head, which drops the number of stored sets sharply. For example, Llama 2 and 3 at 70B and Mistral 7B share down to 8 key-value heads, which cuts the cache by roughly eightfold against full multi-head attention. This is why a recent 70B model can hold a smaller cache than an older 7B one.
There is a more aggressive version called multi-query attention, where every query head shares a single key-value head. It saves the most memory of any head-sharing scheme. However, pushed that far, quality tends to drop, and training grows unstable, so most setups settle on the grouped middle ground as the better trade.
The second attack keeps the heads and compresses what each one stores.
Multi-head latent attention, introduced in the DeepSeek models, projects the keys and values down into a smaller latent representation before caching them, then expands them back when they are read. The savings are large. DeepSeek-V3 holds around 70 kilobytes per token, where comparable grouped-query models sit between 192 and 328. The cost lies in serving, since the compression adds work on every read and pairs awkwardly with some standard attention implementations, so it tends to pay off most once models and contexts grow large enough that cache traffic dominates.
As mentioned, head-sharing and latent attention both require control over the architecture, so they help when we are choosing or training a model. The next attacks work on a model we already have in hand.
Quantization
Quantization goes after the bytes per number.
The keys and values are usually stored at 16 bits each, and quantization rounds them to a smaller format such as 8 bits or 4 bits. Since the bytes-per-number term sits right there in the equation, moving from 16 bits to 8 halves the whole cache, and going to 4 bits halves it again. The appeal is that this applies to a model we already have and skips retraining entirely.
The quality cost depends on how far we push it.
Eight-bit storage often costs well under a percent of accuracy, which puts it within the noise for most workloads. Four-bit storage saves more and starts to show measurable losses on demanding tasks such as multi-needle retrieval, where the model has to pull several specific facts out of a long context.
Specialized methods beat plain rounding because a few numbers in the cache deserve more precision than the rest, though plain rounding already captures most of the gain.
Eviction
Eviction goes after the token count by dropping entries that the model is unlikely to need.
The common approach keeps a window of the most recent tokens, since recent context usually matters most, along with a few tokens from the very start of the sequence. Those opening tokens turn out to play an outsized role. They absorb a large share of attention regardless of what they actually say, acting as anchors that keep the model’s output stable.
See the diagram below:
The trouble with eviction is structural.
Whether a token matters depends on a question that has yet to arrive. A token we drop now can be exactly the one a later part of the generation needs, and once it is gone, the model generates as though that token had been absent the whole time. This shows up on retrieval tasks, where an aggressively trimmed cache handles a casual chat well and then misses a fact buried in the middle of a long document.
More refined schemes score each token’s importance and try to predict which ones are safe to drop, which helps, though the core problem remains.
Serving
Even with the contents of the cache fixed, the way a serving system manages memory leaves a lot on the table, and two techniques help.
The first is paged attention.
Older serving systems reserved one large contiguous block per request, sized for the longest output it might produce. Most requests finished well short of that, leaving the reserved space idle, and the fragmentation added up. However, paged attention borrows an idea from operating systems, which break memory into small fixed-size pages and hand them out on demand. The cache gets split into small blocks that can live anywhere in memory, tracked by a lookup table that maps each request to its blocks. The result is that systems that wasted 60 to 80 percent of cache memory to fragmentation dropped that figure below 4 percent, and throughput climbed by two to three times, all from packing the same data more tightly.
The second technique results from the first. Since the cache lives in shareable blocks, two requests that begin with the same text can point at the same physical blocks while each holds its own private continuation. This is the foundation of prefix caching, and the productized version that the major APIs call prompt caching.
The win is large for any workload that repeats a prefix, such as an agent that sends the same multi-thousand-token system prompt on every call. OpenAI and Anthropic both report cost and latency reductions of 50 to 90 percent on cache hits, with cached tokens billed at a fraction of fresh ones.
However, one thing to note is that sharing cached state across users has opened timing side-channels that can leak information about other people’s prompts, an active concern we will leave aside here.
Tradeoffs
The techniques we’ve looked at look alike in how much memory they save and differ widely in what they ask for in return.
Some are close to free. For example:
Grouped-query attention costs very little quality and has become the safe default, which is why nearly every current model ships with it.
Paged attention and prefix caching barely touch quality, since they change how the cache is stored and shared rather than what it contains.
Other techniques require more. For example:
Quantization is cheap at 8 bits and grows risky as we push toward 4 and below, so the right setting depends on how sensitive the task is.
Latent attention saves most of the architectural options and asks for real engineering effort to serve well.
Eviction frees a large amount of memory and can also lose information that the generation later needs, which makes it a genuine gamble rather than a clean win.
For a short context, the cache is small, and most of these techniques solve a problem that has yet to appear. They earn their place in a long context and high concurrency, where the cache grows into the dominant cost. The right mix follows the workload, with agent loops leaning on reuse and long-document retrieval leaning away from eviction.
Conclusion
The cost of long-context inference comes down to one cache, sized by one short equation.
Since decoding reads the whole cache on every token, the cache is a bandwidth cost as much as a storage one, which is why shrinking it speeds things up. Every optimization we covered deals with a specific aspect of the overall equation or trims the waste around it.
Grouped query and latent attention reduce what each token costs.
Quantization stores each number in fewer bits.
Eviction keeps fewer tokens.
Paged attention and prefix caching manage and share the cache more efficiently.
References:
A noob question. I learned that LLMs are stateless, and here is an entire article on LLM caching. What am I missing? Kindly help me make a distinction.
The 40 GB example is a good reminder that model size is only part of the bill. During generation, the GPU reads the cache for every new token. Long prompts can slow a system down even when the model fits in memory. That is easy to miss when teams benchmark short chats.