OpenRouter's New Routing Algorithm Cuts TTFT 16.9% and Rate-Limit Errors 67% Across 24 Million Requests
OpenRouter overhauled its default provider routing algorithm on October 28, replacing a price-first approach with one that optimizes across four dimensions: effective price, latency, throughput, and error rate. The change applies automatically to all requests using default routing — no code changes required from developers.
How the New Algorithm Works
The previous router sent traffic to the cheapest available provider for each model. The replacement builds a real-time Routing Cost for each endpoint, combining:
- Effective price: derived from a provider’s actual cache hit rate and recent input/output token mix, not list price — eliminating the ability to win traffic with cheap input rates while burying cost in output tokens
- Latency cost: throughput measured end-to-end (TTFT plus full stream time) converted to a cost equivalent at a per-model rate
- Retry cost: reliability errors (provider failures) and quality failures (malformed tool calls) each inflate the expected number of tries needed per good answer
Providers with broken tool calls or benchmark accuracy significantly below the model’s expected score are demoted to low test-traffic until the flag clears.
Session continuity is preserved: the first request in a conversation is routed by the algorithm; subsequent turns are pinned to the same provider to keep the prompt cache warm. If that provider becomes unavailable, the conversation falls back automatically.
A/B Test Results
OpenRouter split over 24 million requests (103 billion input tokens) evenly between the old and new algorithm over a one-week period:
| Measure | Legacy | New | Change |
|---|---|---|---|
| TTFT p50 | 1,007 ms | 837 ms | −16.9% |
| TTFT p95 | 4,601 ms | 3,733 ms | −18.9% |
| Throughput p50 | 62.4 tok/s | 72.6 tok/s | +16.3% |
| Whole request p50 | 4.00 s | 3.24 s | −19.0% |
| Rate-limit errors (429) | 7.1% | 3.8% | −67% relative |
| Requests needing retry | 10.4% | 6.9% | −47% |
| Time in failed attempts | 349 ms | 242 ms | −55% |
| Cost per 1K requests | $1.25 | $1.32 | +6.0% |
The caller-side TTFT p95 fell from 6,032 ms to 4,504 ms (25.3% reduction), a larger gain than the raw endpoint figures because retry overhead is eliminated upstream.
Cost Trade-Off
The 6% cost increase reflects the algorithm occasionally selecting a faster or more reliable provider over the absolute cheapest. OpenRouter describes this as the price of paying for “what a good answer costs” rather than what a token costs.
Developers who prefer the old behavior can set provider.sort: "price" or append :floor to the model slug. Explicit provider.order or provider.only bypass the algorithm entirely. The new routing applies only when no explicit provider preference is set.
Provider Incentives
The algorithm creates a direct market signal: providers that improve latency or reliability automatically win more traffic without negotiating terms. OpenRouter says this is before any provider has had time to optimize for the new scoring — implying the gains should compound over subsequent months as providers adapt.