Prompt Caching Cut Our AI Costs by 70%

We were bleeding money on AI API costs. Then we discovered prompt caching.

The Problem

Our AI customer service bot was handling 100K conversations/month. Each conversation:

4KB system prompt

2KB few-shot examples

1KB user context

Variable user messages

Cost: $50,000/month 💸

The Insight

90% of our tokens were static:

System prompt: Same for everyone

Few-shot examples: Rarely change

User context: Same within a session

We were paying to send the same tokens over and over.

Solution: Prompt Caching

Anthropic's Cache

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic();

const response = await anthropic.messages.create({
model: "claude-3-5-sonnet-20241022",
max_tokens: 1024,
system: [
{
type: "text",
text: systemPrompt, // 4KB of instructions
cache_control: { type: "ephemeral" } // 👈 Cache this
},
{
type: "text",
text: fewShotExamples, // 2KB of examples
cache_control: { type: "ephemeral" } // 👈 Cache this too
}
],
messages: [{ role: "user", content: userMessage }]
});

// Cache hit = 90% cost reduction on cached tokens!

OpenAI's Approach

// OpenAI uses automatic caching for identical prefixes
// Just structure your prompts consistently:

const messages = [
{ role: "system", content: STATIC_SYSTEM_PROMPT },
{ role: "user", content: STATIC_CONTEXT },
{ role: "user", content: dynamicUserInput }, // Only this varies
];

// OpenAI automatically caches the static prefix

Self-Hosted with Semantic Cache

import { SemanticCache } from "ai-cache";

const cache = new SemanticCache({
redis: redisClient,
embedding: openai.embeddings,
threshold: 0.95, // Similarity threshold
});

async function query(prompt: string) {
// Check semantic cache
const cached = await cache.get(prompt);
if (cached) return cached;

// Miss - call API
const response = await llm.complete(prompt);
await cache.set(prompt, response);
return response;
}

Advanced Strategies

1. Prompt Deduplication

// Before: Redundant context in every message
messages.push({ role: "user", content: Context: ${ctx}\n\nQuestion: ${q1} });
messages.push({ role: "user", content: Context: ${ctx}\n\nQuestion: ${q2} });

// After: Context once, questions reference it
messages.push({ role: "system", content: Context: ${ctx} });
messages.push({ role: "user", content: q1 });
messages.push({ role: "user", content: q2 });

2. Response Caching

// Cache entire responses for common queries
const cacheKey = hash(prompt + model + temperature);
const cached = await redis.get(cacheKey);
if (cached) return JSON.parse(cached);

3. Tiered Models

// Use cheaper models for simple queries
const complexity = await classifier.analyze(query);
const model = complexity > 0.7 ? "gpt-4" : "gpt-3.5-turbo";

Results

| Metric | Before | After |
|--------|--------|-------|
| Monthly cost | $50,000 | $15,000 |
| Avg latency | 2.1s | 0.8s |
| Cache hit rate | 0% | 73% |

Key Takeaways

1. Audit your prompts - Find the static parts
2. Structure for caching - Put static content first
3. Monitor hit rates - Optimize based on data
4. Combine strategies - Caching + model routing + response cache

The best AI cost optimization doesn't sacrifice quality. It eliminates waste.