TypeSafe's Jev Matches Claude Opus 5 Within 3.3 Points at 1/22 the Cost on Banking Classification
TypeSafe’s Jev is a decision model: give it an app state and a typed question, it returns a typed answer, a confidence score, and probability distributions across every possible option. OpenRouter’s engineering team ran Jev 1.13 against Claude Opus 5 on 3,080 Banking77 customer-support utterances — 77 intent categories, same one-line description of each intent for both models.
The raw numbers
| Jev 1.13 | Claude Opus 5 | |
|---|---|---|
| Accuracy | 81.0% | 84.4% |
| Macro-F1 | 80.5% | 83.6% |
| Latency p50 | 175 ms | 2,266 ms |
| Latency p99 | 353 ms | 3,835 ms |
| Cost / 1,000 requests | $0.11 | $2.42 |
Jev is 3.3 accuracy points below Opus, 13 times faster at the median, and costs 1/22nd as much. Both models produced zero invalid responses across all 3,080 runs.
One counter-intuitive detail: Jev returned an average of 826 tokens per request — probability distributions across all 77 labels — while Opus returned 17 tokens per request. Jev is cheap despite longer outputs because its per-token rate is low. Opus’s brevity is irrelevant when the per-token cost is high enough.
The cascade is the real finding
Jev outputs a confidence score on every decision. OpenRouter tested what happens when low-confidence Jev responses are routed to Opus:
| Threshold | Jev handles | Accuracy | Cost / 1,000 |
|---|---|---|---|
| Jev only | 100% | 81.0% | $0.11 |
| 0.90 | 76% | 84.0% | $0.69 |
| 0.95 | 68% | 84.2% | $0.88 |
| Opus only | 0% | 84.4% | $2.42 |
At the 0.90 threshold, 76% of requests never reach Opus. Accuracy falls to 84.0% — 0.4 points below full Opus, within confidence intervals. Cost drops to $0.69 per thousand: under 30% of Opus-only pricing.
The score earns its keep as a ranking and routing signal, even though it is not calibrated as a true probability — OpenRouter notes that thresholds should be tuned on your own data rather than treated as absolute probability estimates. Jev reached 96.3% accuracy on utterances where it scored 0.99 or above confidence — 58% of the total. On utterances rated below 0.50 confidence (3.5% of the set), accuracy fell to 29.6%. The model reliably signals when it is uncertain.
The comparison benchmark
OpenRouter selected Claude Opus 5 as the comparison model because it is what OpenRouter users currently spend the most on for classification workloads. The test ran Opus with the 3,750-token system prompt cached across all requests, so Opus was billed at the cheaper cache-read rate rather than list price. Without caching, the list-rate cost would be approximately $19 per thousand requests — making the Jev cost advantage roughly 170-to-one rather than 22-to-one.
Banking77 is a controlled benchmark, not production data. Real-world accuracy will vary with taxonomy complexity and the similarity of intents in the deployment. But the routing logic — run the specialist first, escalate when it reports low confidence — is not benchmark-specific. The mechanism generalizes to any classification workload where a confidence score is available.
The practical verdict from the numbers: if three accuracy points matter more than a 22x cost multiplier, use Opus. For high-volume workloads where accuracy within 0.4 points of Opus is acceptable, the cascade at 0.90 gets there at $0.69 per thousand.