← Back to feed

OpenRouter’s October newsletter, titled “Cache is Crucial,” puts numbers on how dramatically agentic inference infrastructure changed in the first nine months of 2026. The data covers prompt caching across 35 inference providers and reveals a gap in January that most of the industry has since closed.

In January 2026: two of 35 inference providers served agentic traffic at cache rates above 90%, and both were operating at small volume. The median provider cached 44% of agentic prompt tokens. Even the two largest players, Google and Anthropic, sat between 65% and 80%.

By September 2026, the situation had shifted substantially — enough that OpenRouter credits “all layers of the stack, from the OpenRouter algorithm to the model makers to the inference providers” with driving the acceleration.

Why Cache Rates Matter for Agentic Economics

Agentic workloads are structurally cache-friendly. Every step in a multi-turn agent loop re-sends the system prompt, prior conversation context, and tool definitions. If those tokens are served from cache rather than computed fresh, the inference cost drops sharply — typically to 10-25% of full-compute pricing at providers who offer prompt caching discounts.

A median cache rate of 44% in January meant that more than half of agentic prompt tokens were being computed from scratch on each request. At the token volumes that production agentic workloads generate — millions of prompt tokens per task, across thousands of concurrent tasks — that is a significant cost multiplier.

Moving the median from 44% toward 90%+ is not a trivial optimization. It requires coordination across three layers: the routing layer (which must send requests to providers who already hold the prefix in cache), the model providers (who must expose stable cache keys and offer pricing incentives for cache hits), and the inference providers (who must allocate KV cache memory efficiently and maintain it across requests).

What Changed

The acceleration reflects a series of changes that stacked over 2026:

Anthropic and OpenAI both extended prompt caching more aggressively, with Sonnet 5.5 cache reads priced at $0.20 per million tokens and GPT-6 family cache pricing structured to reward long-running agentic sessions.

Inference providers — including those hosted on Nvidia hardware at hyperscaler scale — began dedicating KV cache memory pools specifically to agentic prefix patterns, rather than treating each request as independent.

OpenRouter’s routing algorithm updated to factor cache residency into provider selection, not just latency and cost. A request routed to a provider that already holds the prefix in cache is materially cheaper than the same request routed to a provider that will compute it fresh.

The Compounding Effect

At 90%+ cache rates, the effective cost of an agentic workload drops to roughly what the non-cached marginal tokens cost — the parts of each request that are genuinely new. For long-running agents with stable system prompts and growing context windows, that marginal fraction can be 10-20% of total tokens sent.

The implication: the real cost of running frontier models for agentic tasks has fallen faster than headline pricing changes suggest. Models got more expensive per token in some cases during 2026 as capabilities improved, but the effective per-task cost moved in the opposite direction as caching infrastructure matured.

OpenRouter’s data covers the provider-level view. Individual enterprise deployments with stable agent configurations — fixed system prompts, predictable tool schemas — will see cache rates at the high end of what providers report, since their traffic is maximally cache-friendly by structure.

The full newsletter is at openrouter.ai/data/newsletter/cache-is-crucial.