The distinction between reusing a KV prefix and reusing an answer is especially clear here.
In a RAG pipeline, semantic-cache invalidation becomes a provenance problem: a cached answer can depend on several retrieved chunks, each with its own version and access scope. How would you record that dependency set and test that a changed or revoked chunk cannot leak through a still-similar cache hit without flushing the entire cache?
The semantic miss path deserves its own latency budget: embedding, lookup, then fresh generation can be slower than skipping the cache entirely. I'd put a short deadline on that lookup and fall back to normal generation when the cache is unavailable, while retaining the same scope and freshness checks for any accepted hit. Measure tail latency separately for hits, misses, and lookup timeouts. Otherwise a cheaper average request can hide a worse experience for the users who miss.
I honestly don't think the cancelled-subscription case is a huge problem in practice. A decent setup routes anything that triggers an action around the cache, so only true repeat questions ever get a stored answer. Telling a new request apart from an old one is pretty easy at that routing step.
the placement rule is the part most people will get wrong in practice. in an agent loop every turn appends new tool output, so the shared prefix is only the system prompt and the earliest messages. worth structuring prompts with the stable material first and the volatile stuff last, otherwise the prefix cache dies after the first tool call.
The cancellation line is the cache entry I would refuse to store. A sentence that says the subscription was cancelled can be the nearest neighbor of the next similar question and still be false, because the cache never ran the side effect. I would keep prompt caching for the shared manual, keyed on that manual's revision so an edit misses the prefix, and allow a semantic entry only when the original request did not call a tool. A reused KV prefix and a reused "it is done" message are not the same kind of savings.
Tool schemas built from a Python set will quietly break this, since string hashing changes between processes and the tool order shifts with it. Every request then misses the prefix and pays full input price. Charting cached_tokens (cache_read_input_tokens on Anthropic) per request shows it the day it starts.
The point that a 0.92 similarity score isn't a 92% chance the answer fits is the one most teams learn the hard way. On scoping, I've found tenant, product version, language and source-data version work better as hard partitions of the index than as filters applied after the top-k search, since a post-filter can leave zero valid candidates and quietly turn a real hit into a miss. I'm also curious about the interaction between the two caches: a semantic hit skips the LLM call, so it also stops refreshing the provider's prefix cache. A high semantic hit rate could let the prefix cache expire and make the remaining misses more expensive. Have you seen teams tune for that?
The distinction between reusing a KV prefix and reusing an answer is especially clear here.
In a RAG pipeline, semantic-cache invalidation becomes a provenance problem: a cached answer can depend on several retrieved chunks, each with its own version and access scope. How would you record that dependency set and test that a changed or revoked chunk cannot leak through a still-similar cache hit without flushing the entire cache?
The semantic miss path deserves its own latency budget: embedding, lookup, then fresh generation can be slower than skipping the cache entirely. I'd put a short deadline on that lookup and fall back to normal generation when the cache is unavailable, while retaining the same scope and freshness checks for any accepted hit. Measure tail latency separately for hits, misses, and lookup timeouts. Otherwise a cheaper average request can hide a worse experience for the users who miss.
I honestly don't think the cancelled-subscription case is a huge problem in practice. A decent setup routes anything that triggers an action around the cache, so only true repeat questions ever get a stored answer. Telling a new request apart from an old one is pretty easy at that routing step.
the placement rule is the part most people will get wrong in practice. in an agent loop every turn appends new tool output, so the shared prefix is only the system prompt and the earliest messages. worth structuring prompts with the stable material first and the volatile stuff last, otherwise the prefix cache dies after the first tool call.
What is the difference between embedding and KV state ?
The cancellation line is the cache entry I would refuse to store. A sentence that says the subscription was cancelled can be the nearest neighbor of the next similar question and still be false, because the cache never ran the side effect. I would keep prompt caching for the shared manual, keyed on that manual's revision so an edit misses the prefix, and allow a semantic entry only when the original request did not call a tool. A reused KV prefix and a reused "it is done" message are not the same kind of savings.
Tool schemas built from a Python set will quietly break this, since string hashing changes between processes and the tool order shifts with it. Every request then misses the prefix and pays full input price. Charting cached_tokens (cache_read_input_tokens on Anthropic) per request shows it the day it starts.
The point that a 0.92 similarity score isn't a 92% chance the answer fits is the one most teams learn the hard way. On scoping, I've found tenant, product version, language and source-data version work better as hard partitions of the index than as filters applied after the top-k search, since a post-filter can leave zero valid candidates and quietly turn a real hit into a miss. I'm also curious about the interaction between the two caches: a semantic hit skips the LLM call, so it also stops refreshing the provider's prefix cache. A high semantic hit rate could let the prefix cache expire and make the remaining misses more expensive. Have you seen teams tune for that?