PolyAI's Raven 3.5 Beats GPT-5 and Claude Sonnet 4.6 on All 4 Customer Service Benchmarks at Under 300ms
The specialist-versus-generalist debate in enterprise AI has a new data point. PolyAI’s Raven 3.5, a model purpose-built for customer service phone calls, beats GPT-5 and Claude Sonnet 4.6 on all four of PolyAI’s internal customer service benchmarks — and does it at under 300 milliseconds latency.
Key Numbers
- 4/4 benchmarks beat vs GPT-5 and Claude Sonnet 4.6
- < 300ms latency end-to-end on voice calls
- 23 languages with near-perfect language adherence
- ~50% of turns use auto-reasoning — averaging under 40 tokens of internal chain-of-thought, adding a few hundred milliseconds only on complex turns
The Training Recipe
Raven 3.5 uses a GRPO + DPO fusion — reinforcement learning via Group Relative Policy Optimization combined with Direct Preference Optimization against curated examples of what good support responses look like. The training data is millions of anonymized real customer service conversations, not synthetic prompts or internet text.
The result: the model has deeply internalized what live phone support actually demands — handling interruptions, tracking multi-turn goal state, language switching mid-conversation, and following complex multi-step instructions across domains like banking, healthcare, retail, and hospitality.
Where general models trip: they struggle when the user’s source language and the system prompt are in English but the agent is instructed to respond in Spanish. Raven 3.5 handles this natively because it was trained specifically to, not as an afterthought in a general-purpose multilingual pretraining run.
Auto-Reasoning Without Latency Penalty
Raven 3.5 includes latency-aware auto-reasoning. The model decides at runtime whether a turn needs deliberation — it reasons on roughly half of turns, usually producing fewer than 40 tokens of internal chain-of-thought in a few hundred milliseconds. Complex requests like “book me something three Tuesdays from now, but not if it conflicts with the holidays” get reasoning. “What are your opening hours?” does not.
This is a real constraint: frontier reasoning models like o3 or Claude Opus 4.7 with extended thinking cannot be used in live phone calls because the deliberation latency breaks conversational rhythm. PolyAI’s approach trains the reasoning decision into the model itself rather than imposing it externally.
New Product Launches
Alongside the model, PolyAI is launching two products:
ADK (Agent Development Kit) — A code-first SDK for building production voice agents from any IDE. Integrates directly with PolyAI’s Agent Studio. Designed for developers who want full control over deployment, versioning, and testing workflows rather than a no-code path.
PolyPhone — A tool that reads a website (FAQs, product pages, knowledge base) and generates a deployable voice agent in approximately 10 minutes. The agent can be embedded on any webpage without telephony infrastructure setup. Target user: a marketing or product team that wants a website to be talkable without involving an AI engineering team.
The Thesis
PolyAI has held this position since building their first model: a domain expert trained with purpose will outperform a generalist at the job it was built for. The Raven 3.5 results are the strongest version of that argument yet — beating models that are probably 100x larger by parameter count, on tasks that matter for the deployment PolyAI is targeting.
The counterargument is scope: Raven 3.5 is exceptional at phone-based customer service and will degrade outside of it. GPT-5 handles anything. Enterprise buyers paying $5–10/hour for contact center agents may find that tradeoff acceptable. Buyers who want one model for everything will not.
Raven 3.5 is available now. ADK and PolyPhone are launching in limited access.