Why Token-Reduction Percentages Lie About AI Savings

If you're evaluating ways to cut your team's AI coding bill, you'll hear a lot of impressive percentages. "Cut tokens 40%." "Reduce context by half." They sound like savings. Often, they aren't — at least not the kind that shows up on your invoice.

Here's why token-based claims mislead, and what to measure instead.

Tokens are not dollars

The core problem: a token-reduction percentage counts tokens, not money. Two things break the link between the two.

1. Prompt caching

Most model providers — across Bedrock, Azure AI Foundry, and Vertex — support prompt caching. Cached input tokens are billed at a steep discount compared to uncached ones.

So consider a "savings" that removes a large block of context from a prompt. If that context was already cached, it was costing you a fraction of the headline rate. Cutting it shows up as a big token reduction — and a tiny dollar reduction. The percentage looks great; the invoice barely moves.

2. Retries

AI coding tools retry. Calls fail, time out, get repeated, or get re-run after a bad result. Those retries cost money, and they rarely appear in a clean "tokens saved" calculation. A measure that only looks at the tokens in a successful happy-path request misses a real chunk of actual spend.

Put caching and retries together and a token-reduction percentage can be off from the real dollar impact by a wide margin — almost always in the optimistic direction.

What to measure instead: net-of-cache dollars

The number a manager actually cares about is simple: how much less are we paying? To answer that honestly, measure:

• Dollars, not tokens. Start from billed spend.
• Net of cache. Reflect the cache discount that was actually applied, so you're comparing what you really paid.
• Including retries. Count the repeated and failed calls that still cost money.
• Broken down by team and task type, so you know where the savings came from and whether they hold up.

If a savings claim can't be tied back to the invoice, it isn't a savings claim — it's a vanity metric.

Keep the scorekeeper separate from the player

There's one more trap. If the same system both decides how to optimize (routing, compression) and reports how much it saved, it's grading its own homework. It will tend to estimate savings from its own decisions rather than from what was actually billed.

The more trustworthy pattern is independent measurement: a layer that observes real billed spend, separate from the logic making the optimization choices. When the scorekeeper and the player are different, the number means something.

A simple test for any AI cost tool

Before you trust a savings figure, ask:

1. Is this in dollars, or tokens/percentages?
2. Does it account for prompt caching?
3. Does it include retries and failed calls?
4. Can I tie it back to my actual invoice?
5. Is the thing measuring savings independent from the thing making the changes?

If the answers are vague, assume the real number is smaller.

How Tokor approaches this

We built measurement into the foundation of Tokor, KorPro's AI cost optimization product, because we kept seeing token-percentage claims that didn't survive contact with the invoice.

Tokor is a self-hosted model router and measurement layer for AI coding tools (starting with Claude Code) on Bedrock, Azure Foundry, and Vertex. It routes each task to the cheapest capable model — but just as importantly, it reports net-of-cache dollars saved, with the measurement layer kept independent from the routing logic. You can start in shadow mode and watch the real number before changing anything.

Tokor is in early access. If you run 20+ developers on AI coding tools and want savings you can actually defend to finance, apply to the design partner program.

Related: How to Reduce AI Coding Costs Across a Developer Team.