Semantic Caching for LLM APIs: How to Stop Paying for the Same Question Twice

Ask a support bot "how do I reset my password" and then, five minutes later, ask "I forgot my password, can you help me log back in" and most LLM-backed apps treat those as two unrelated requests. Same intent, same correct answer, two full model calls. Exact-match caching can't catch this because the strings don't match, and prefix caching can't catch it either, because the part that differs is the part right after the shared system prompt - the part prefix caching was never built to compare. Semantic caching is the mechanism built specifically for this gap, and it works on a genuinely different principle than the caching most teams reach for first.

What semantic caching actually compares

A normal cache checks whether it has seen this exact input before. A semantic cache checks whether it has seen an input that means the same thing. Every incoming query gets converted to an embedding vector, and that vector gets compared against the vectors of past queries sitting in a vector index, using cosine similarity rather than string equality. If the closest match clears a similarity threshold - implementations typically land somewhere between 0.92 and 0.97, per a 2026 review of semantic caching architectures on arXiv - the cached response comes back immediately and the LLM never gets called. This is the same nearest-neighbor search that powers retrieval-augmented generation, just pointed at a store of past questions instead of a knowledge base, which is why it's usually built on the same vector database infrastructure a RAG pipeline already uses.

Semantic caching vs prompt caching: different layers, different savings

It's easy to conflate this with prompt caching, but they solve different problems and the difference matters for where you spend engineering time. Prompt caching, as Redis's own comparison explains, stores the model's internal computation for a repeated prompt prefix, so the provider skips redundant prefill work and charges a fraction of the input-token price on the cached portion - Anthropic's cache reads run at roughly a tenth of standard input pricing, as covered in our prompt caching guide. It only fires when requests share a literal prefix, and it only discounts input tokens; the model still runs and you still pay for output.

Semantic caching works upstream of the model entirely. On a hit, there is no model call at all - no input cost, no output cost, no latency beyond the vector lookup. The tradeoff is that it only fires on meaning-level repeats, and unlike prompt caching, it can return a wrong answer if the similarity match is loose, which prompt caching structurally cannot do since it never skips the model's actual reasoning. Most production systems that need both run them as separate layers: semantic cache first to catch repeated questions outright, prompt caching underneath for everything that falls through, since a stable system prompt still benefits even on a genuinely new question.

Building one: GPTCache and the open-source path

You don't need a vendor contract to try this. GPTCache, built by Zilliz and fully open source, is the reference implementation most teams start from - it wraps an embedding model, a vector store, and a similarity evaluator behind a small API that sits in front of your existing OpenAI, Anthropic, or Ollama calls. It supports Milvus, Faiss, Redis, and Qdrant as interchangeable backends, so if you already run one of those for RAG, the semantic cache is a second collection in infrastructure you're already operating, not a new system to maintain. Redis itself documents the same pattern using its own vector search module directly, without GPTCache in the loop, if you'd rather keep one fewer dependency. Neither path requires a paid caching product - LangCache and similar managed offerings exist for teams that don't want to own the threshold-tuning and eviction logic themselves, but the open-source route is a real, complete option, not a crippled trial version of it.

The failure mode nobody mentions: confidently wrong cache hits

Here's the part that gets skipped in most "add semantic caching" writeups: a loose threshold doesn't just cost you a few unnecessary cache hits, it can return a plausible, fluent, wrong answer with total confidence. "What's the refund window on the Basic plan" and "what's the refund window on the Pro plan" sit close together in embedding space - they're the same sentence structure with one word swapped - but if your plans have different refund terms, a 0.94-similarity match hands the user the wrong policy, and nothing in the response signals that anything went wrong. This isn't a hypothetical edge case; it's the direct consequence of using semantic distance as a proxy for factual equivalence, and the two aren't the same thing. The practical fix is treating the threshold as a per-use-case tuning parameter, not a default you copy from a blog post: start conservative (0.97+), log every near-miss the cache almost served, and only loosen it once you've confirmed a lower threshold isn't quietly swapping in wrong answers for numeric details, prices, or anything plan- or account-specific.

Where it earns its keep, and where it doesn't

Semantic caching pays off fastest in FAQ-style support bots, internal knowledge assistants, and any RAG app where a large fraction of real user questions are paraphrases of a much smaller set of underlying questions - which is most customer-facing deployments, since people ask the same handful of things in dozens of different phrasings. It's the wrong tool for personalized responses (where two similar-looking questions legitimately need different answers because the account or context differs), code generation (where near-identical prompts often need genuinely distinct outputs), and anything where a stale or slightly-off cached answer creates real risk rather than minor annoyance.

Building this into a real product

Getting semantic caching right isn't really about picking GPTCache versus rolling your own Redis lookup - both work. It's about the same judgment call that shows up everywhere in production AI systems: knowing which layer of your stack should trade a small chance of being wrong for a large reduction in cost and latency, and instrumenting it well enough to catch the difference when that trade goes bad. AI Product Development Bootcamp covers this alongside the retrieval and evaluation patterns from our RAG pipeline evaluation guide - the same instinct for measuring what a shortcut costs you before you ship it.