Our LLM API bill was growing 30% month-over-month. Traffic was increasing, but not that fast. When I analyzed our query logs, I found the real problem: Users ask the same questions in different ways.
"What's your return policy?," "How do I return something?", and "Can I get a refund?" were all hitting our LLM separately, generating nearly identical responses, each incurring full API costs.
Exact-match caching, the obvious first solution, captured only 18% of these redundant calls. The same semantic question, phrased differently, bypassed the cache entirely.
So, I implemented semantic caching based on what queries mean, not how they're worded. After implementing it, our cache hit rate increased to 67%, reducing LLM API costs by 73%. But getting there requires solving problems that naive implementations miss.
Traditional caching uses query text as the cache key. This works when queries are identical:
# Exact-match caching
cache_key = hash(query_text)
if cache_key in cache:
return cache[cache_key]
But users don't phrase questions identically. My analysis of 100,000 production queries found:
Only 18% were exact duplicates of previous queries
47% were semantically similar to previous queries (same intent, different wording)
35% were genuinely novel queries
That 47% represented massive cost savings we were missing. Each semantically-similar query triggered a full LLM call, generating a response nearly identical to one we'd already computed.
Semantic caching replaces text-based keys with embedding-based similarity lookup:
class SemanticCache:
def __init__(self, embedding_model, similarity_threshold=0.92):
self.embedding_model = embedding_model
self.threshold = similarity_threshold
self.vector_store = VectorStore() # FAISS, Pinecone, etc.
self.response_store = ResponseStore() # Redis, DynamoDB, etc.
def get(self, query: str) -> Optional[str]:
"""Return cached response if semantically similar query exists."""
query_embedding = self.embedding_model.encode(query)
# Find most similar cached query
matches = self.vector_store.search(query_embedding, top_k=1)
if matches and matches[0].similarity >= self.threshold:
cache_id = matches[0].id
return self.response_store.get(cache_id)
return None
def set(self, query: str, response: str):
"""Cache query-response pair."""
query_embedding = self.embedding_model.encode(query)
cache_id = generate_id()
self.vector_store.add(cache_id, query_embedding)
self.response_store.set(cache_id, {
'query': query,
'response': response,
'timestamp': datetime.utcnow()
})
The key insight: Instead of hashing query text, I embed queries into vector space and find cached queries within a similarity threshold.
The similarity threshold is the critical parameter. Set it too high, and you miss valid cache hits. Set it too low, and you return wrong responses.
Our initial threshold of 0.85 seemed reasonable; 85% similar should be "the same question," right?
Wrong. At 0.85, we got cache hits like:
Query: "How do I cancel my subscription?"
Cached: "How do I cancel my order?"
Similarity: 0.87
These are different questions with different answers. Returning the cached response would be incorrect.
I discovered that optimal thresholds vary by query type:
Query type | Optimal threshold | Rationale |
FAQ-style questions | 0.94 | High precision needed; wrong answers damage trust |
Product searches | 0.88 | More tolerance for near-matches |
Support queries | 0.92 | Balance between coverage and accuracy |
Transactional queries | 0.97 | Very low tolerance for errors |
I implemented query-type-specific thresholds:
class AdaptiveSemanticCache:
def __init__(self):
self.thresholds = {
'faq': 0.94,
'search': 0.88,
'support': 0.92,
'transactional': 0.97,
'default': 0.92
}
self.query_classifier = QueryClassifier()
def get_threshold(self, query: str) -> float:
query_type = self.query_classifier.classify(query)
return self.thresholds.get(query_type, self.thresholds['default'])
def get(self, query: str) -> Optional[str]:
threshold = self.get_threshold(query)
query_embedding = self.embedding_model.encode(query)
matches = self.vector_store.search(query_embedding, top_k=1)
if matches and matches[0].similarity >= threshold:
return self.response_store.get(matches[0].id)
return None
I couldn't tune thresholds blindly. I needed ground truth on which query pairs were actually "the same."
Our methodology:
Step 1: Sample query pairs. I sampled 5,000 query pairs at various similarity levels (0.80-0.99).
Step 2: Human labeling. Annotators labeled each pair as "same intent" or "different intent." I used three annotators per pair and took a majority vote.
Step 3: Compute precision/recall curves. For each threshold, we computed:
Precision: Of cache hits, what fraction had the same intent?
Recall: Of same-intent pairs, what fraction did we cache-hit?
def compute_precision_recall(pairs, labels, threshold):
"""Compute precision and recall at given similarity threshold."""
predictions = [1 if pair.similarity >= threshold else 0 for pair in pairs]
true_positives = sum(1 for p, l in zip(predictions, labels) if p == 1 and l == 1)
false_positives = sum(1 for p, l in zip(predictions, labels) if p == 1 and l == 0)
false_negatives = sum(1 for p, l in zip(predictions, labels) if p == 0 and l == 1)
precision = true_positives / (true_positives + false_positives) if (true_positives + false_positives) > 0 else 0
recall = true_positives / (true_positives + false_negatives) if (true_positives + false_negatives) > 0 else 0
return precision, recall
Step 4: Select threshold based on cost of errors. For FAQ queries where wrong answers damage trust, I optimized for precision (0.94 threshold gave 98% precision). For search queries where missing a cache hit just costs money, I optimized for recall (0.88 threshold).
Semantic caching adds latency: You must embed the query and search the vector store before knowing whether to call the LLM.
Our measurements:
Operation | Latency (p50) | Latency (p99) |
Query embedding | 12ms | 28ms |
Vector search | 8ms | 19ms |
Total cache lookup | 20ms | 47ms |
LLM API call | 850ms | 2400ms |
The 20ms overhead is negligible compared to the 850ms LLM call we avoid on cache hits. Even at p99, the 47ms overhead is acceptable.
However, cache misses now take 20ms longer than before (embedding + search + LLM call). At our 67% hit rate, the math works out favorably:
Before: 100% of queries × 850ms = 850ms average
After: (33% × 870ms) + (67% × 20ms) = 287ms + 13ms = 300ms average
Net latency improvement of 65% alongside the cost reduction.
Cached responses go stale. Product information changes, policies update and yesterday's correct answer becomes today's wrong answer.
I implemented three invalidation strategies:
Simple expiration based on content type:
TTL_BY_CONTENT_TYPE = {
'pricing': timedelta(hours=4), # Changes frequently
'policy': timedelta(days=7), # Changes rarely
'product_info': timedelta(days=1), # Daily refresh
'general_faq': timedelta(days=14), # Very stable
}
When underlying data changes, invalidate related cache entries:
class CacheInvalidator:
def on_content_update(self, content_id: str, content_type: str):
"""Invalidate cache entries related to updated content."""
# Find cached queries that referenced this content
affected_queries = self.find_queries_referencing(content_id)
for query_id in affected_queries:
self.cache.invalidate(query_id)
self.log_invalidation(content_id, len(affected_queries))
For responses that might become stale without explicit events, I implemented periodic freshness checks:
def check_freshness(self, cached_response: dict) -> bool:
"""Verify cached response is still valid."""
# Re-run the query against current data
fresh_response = self.generate_response(cached_response['query'])
# Compare semantic similarity of responses
cached_embedding = self.embed(cached_response['response'])
fresh_embedding = self.embed(fresh_response)
similarity = cosine_similarity(cached_embedding, fresh_embedding)
# If responses diverged significantly, invalidate
if similarity < 0.90:
self.cache.invalidate(cached_response['id'])
return False
return True
We run freshness checks on a sample of cached entries daily, catching staleness that TTL and event-based invalidation miss.
After three months in production:
Metric | Before | After | Change |
Cache hit rate | 18% | 67% | +272% |
LLM API costs | $47K/month | $12.7K/month | -73% |
Average latency | 850ms | 300ms | -65% |
False-positive rate | N/A | 0.8% | — |
Customer complaints (wrong answers) | Baseline | +0.3% | Minimal increase |
The 0.8% false-positive rate (queries where we returned a cached response that was semantically incorrect) was within acceptable bounds. These cases occurred primarily at the boundaries of our threshold, where similarity was just above the cutoff but intent differed slightly.
Don't use a single global threshold. Different query types have different tolerance for errors. Tune thresholds per category.
Don't skip the embedding step on cache hits. You might be tempted to skip embedding overhead when returning cached responses, but you need the embedding for cache key generation. The overhead is unavoidable.
Don't forget invalidation. Semantic caching without invalidation strategy leads to stale responses that erode user trust. Build invalidation from day one.
Don't cache everything. Some queries shouldn't be cached: Personalized responses, time-sensitive information, transactional confirmations. Build exclusion rules.
def should_cache(self, query: str, response: str) -> bool:
"""Determine if response should be cached.""
# Don't cache personalized responses
if self.contains_personal_info(response):
return False
# Don't cache time-sensitive information
if self.is_time_sensitive(query):
return False
# Don't cache transactional confirmations
if self.is_transactional(query):
return False
return True
Semantic caching is a practical pattern for LLM cost control that captures redundancy exact-match caching misses. The key challenges are threshold tuning (use query-type-specific thresholds based on precision/recall analysis) and cache invalidation (combine TTL, event-based and staleness detection).
At 73% cost reduction, this was our highest-ROI optimization for production LLM systems. The implementation complexity is moderate, but the threshold tuning requires careful attention to avoid quality degradation.
Sreenivasa Reddy Hulebeedu Reddy is a lead software engineer.