← Back to feed

OpenRouter has published live benchmarks for web search agent configurations, covering every combination of engine, model, and search depth it supports. The data is available at openrouter.ai/benchmarks and updates continuously as new results come in.

The headline findings are counterintuitive for teams that have spent weeks agonising over engine selection.

Search Budget Is the Dominant Variable

The single biggest lever on quality is how much search you give the agent — how many turns it can run, how deep it can go per turn. Switching from a shallow single-turn configuration to a multi-turn deep configuration moves quality more than switching from Exa to Perplexity does, or from one model to another within the same tier.

This matters because search budget is usually the easiest variable to tune after deployment. Swapping engines requires API changes. Adding a search turn is a parameter.

Model Matters More Than Engine

Across all configurations benchmarked, the model doing the synthesis outperforms the engine choice as a quality predictor. An agent running on a frontier model with a mid-tier search engine will typically outperform the same task run on a mid-tier model with a top-tier engine.

OpenRouter tests engines including Exa, Parallel, and Perplexity, alongside the native search built into labs that provide it (OpenAI, Google). The spread between engines on the same model is smaller than the spread between models on the same engine.

Failure Rate Drives Your Worst-Case Cost

The second counterintuitive finding: overall cost is not primarily determined by successful task completions. It is determined by failure rate. Failed search attempts still consume tokens — the agent queries, reads, synthesises, and then fails to produce a useful result. Teams budgeting based on successful-task token counts will systematically underestimate actual spend.

The practical implication: reduce failure rate before reducing search depth. A more reliable shallow configuration is cheaper than an ambitious deep configuration that fails 20% of the time.

What the Benchmarks Cover

The benchmark surface is:

• Engine: Exa, Parallel, Perplexity, native (OpenAI/Google)
• Model: Full OpenRouter model catalogue
• Search depth: Single-turn, multi-turn, deep research modes

Results are stratified by task type — factual recall, multi-step research, real-time data retrieval — since the optimal configuration differs by workload. A customer support agent doing one-shot fact lookups has a different optimal configuration than a research agent building a literature synthesis.

Why It Matters Now

Web search is now standard infrastructure for production agents. Most serious deployments have moved past the question of whether to include search and are trying to optimise which configuration to run. OpenRouter’s benchmarks give teams an empirical baseline to work from rather than running their own ablations from scratch.

The benchmarks are live — results update as OpenRouter adds new engines and models. Teams can submit their own workloads to test against their specific task distribution.