How Cloudflare Cut AI Inference Costs 68%: A Practical Lesson for Every Team
Your AI tool is delivering results. Then one month the invoice comes in 30% higher than the last, and you haven’t changed a thing about how you’re using it.
That’s what happened to Cloudflare. In April, their internal AI code reviewer averaged $1.19 per run. By August, without any major architectural changes or new features, that figure had climbed to $2.46. Models got more capable over those months, engineers started submitting larger and more complex code changes, and the cost crept upward quietly until someone looked at the numbers.
A focused optimization effort brought the cost back down to $0.77 per review. That’s a 68% reduction. At the same time, median review time dropped from four minutes and nine seconds to one minute and thirty-two seconds. Less expensive and nearly three times faster.
How they got there is the instructive part.
About 40% of the savings came from improvements to the reviewer’s own code. The larger portion came from smarter model routing and a fix to how the system handled prompt caching.
Prompt caching is a technique where parts of a conversation that stay the same between AI requests get stored and reused rather than reprocessed from scratch. Reading back a cached result costs roughly one-tenth of what processing those same tokens fresh would cost. Cloudflare discovered their reviewer was only hitting the cache on the very first request between 14% and 26% of the time. It should have been nearly 100%. The culprit was a small formatting issue in how the system prompt was constructed — a timestamp was placed before the cache breakpoint, which meant each new session inadvertently invalidated the prefix it was trying to reuse. Fixing that one structural issue delivered a significant cost drop on its own.
The second lever was reducing redundant tool calls. AI agents in agentic workflows — multi-step processes where the AI makes a series of decisions and actions autonomously — tend to re-check work they’ve already completed. Each unnecessary call adds tokens. Each token costs money. Cutting those redundant calls was straightforward once the team had clear visibility into where the tokens were actually going.
Third, they applied different models to different tasks. Not every step in a code review requires the most powerful and expensive model available. Lighter models handled the lower-stakes work. The heavy-duty model was reserved for decisions that actually needed it.
The bigger lesson here is one that applies to any business running AI tools. Inference costs — the fees charged each time an AI model processes a request — don’t grow only when you add features. They drift upward as models improve, as usage patterns shift, and as the complexity of what you’re feeding the AI increases. Most teams don’t notice until the bill lands.
Treat AI running costs the same way you’d treat any infrastructure spend. Set a baseline. Track it weekly. Investigate when the numbers move unexpectedly. The optimization techniques Cloudflare applied — prompt caching, model routing, reducing unnecessary calls — are available to any team willing to look carefully at where the tokens are going.
Want to explore how AI cost optimization could benefit your business? Let’s talk.