Your LLM Bill Jumped After You Added Context: Find the Cache Miss Before You Downgrade the Model
If your LLM API spend climbed after you added retrieval, a longer system prompt, or tool definitions, the cause is almost always input tokens being reprocessed at full price on every request — not the model you picked.
This article explains why LLM API costs may increase after adding context, system prompts, or tool definitions, and provides strategies to optimize costs by improving prompt caching and using batch processing.
What happened
The article discusses how LLM API costs can unexpectedly increase after adding context, system prompts, or tool definitions. It explains that the issue is often due to input tokens being reprocessed at full price, not the model itself. The article provides guidance on identifying and fixing caching issues, optimizing prompt structure, and using batch processing for cost savings.
Why it matters
Understanding and optimizing prompt caching is critical for reducing LLM costs, especially for applications with repeated requests. Proper cache management and batch processing can significantly lower expenses without sacrificing performance, making these strategies essential for developers and businesses using LLMs at scale.
- Check cache fields in the usage object to identify caching issues.
- Place cache breakpoints at the end of what repeats, not the end of the prompt.
- Use batch endpoints for non-interactive workloads to reduce costs.
- Avoid downgrading models until all free optimizations are exhausted.
- Log and analyze token usage to identify and fix caching bugs.
The bigger picture
Improving prompt caching and using batch processing can lead to substantial cost savings, especially for applications with repeated requests or non-interactive workloads. These optimizations help reduce unnecessary reprocessing of input tokens and improve overall efficiency.
The article emphasizes the importance of measuring and optimizing prompt caching, using batch processing for cost savings, and only considering model downgrades after exhausting free optimizations. These strategies help reduce LLM costs without compromising quality or performance.
Technical details
- Input tokens are split into uncached, cache creation, and cache read tokens.
- Prompt caching is a prefix match over rendered bytes, with changes invalidating the cache.
- Batch endpoints offer asynchronous processing at half the standard per-token rate.
- Caching is model-scoped, and cache entries are only readable after the first response starts streaming.
FAQ
Why is cache_read_input_tokens always 0?
Something in the prefix changes between requests, invalidating the cache. Log two consecutive request bodies and diff them to find the invalidator.
Does prompt caching reduce latency or just cost?
Both: cached prefix tokens are not reprocessed, reducing latency, and cost is also reduced.
Is the LLM batch API worth it for a small app?
Yes, for any job a user isn't waiting on — it offers roughly half the price for the same model and prompt.
Should I route to a smaller model?
Only after exhausting free optimizations like caching and batch processing, as it may compromise quality and cache reuse.
How can I measure LLM costs effectively?
Log input_tokens, cache writes, and cache reads per request, and treat a zero read rate on a repeated prefix as a bug.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by Dev.to AI on Oct 4, 2026.
Comments
Sign in to join the discussion.