← Back to feed

Arena.ai updated its Agent Arena signal leaderboard on October 12 with two changes to how bash failures are measured: a new LLM-based failure detector and a revised scoring formula. The metric, formerly called Bash Recovery, is now named Bash Reliability.

What Was Wrong With the Old System

The previous system used a fixed set of rules to decide whether a bash call failed because of an agent mistake. According to Arena.ai, those rules had two systematic failure modes: they flagged correct test assertions and empty search results as failures, and they missed real errors when those errors were hidden by piped commands. In both cases, the score misrepresented what the agent actually did.

How the LLM Judge Works

The replacement is a single LLM judge that looks at each bash call in context. For each call it evaluates:

• The purpose of the command — what the agent was trying to accomplish
• The outcome — what the shell actually returned
• The cause of any error — and whether the error was a genuine agent mistake
• Self-correction — whether the agent identified and fixed the error within the same call

If an error was caught and resolved before control returned, the call is not penalised. The prior rules had no equivalent mechanism: an error in a piped command counted as a failure regardless of whether the agent handled it.

Scoring Changes

Arena.ai did not publish the full updated scoring formula in the changelog, but confirmed that both the failure detection method and the score derivation changed together. The rename from Bash Recovery to Bash Reliability reflects the shift from measuring error rate to measuring overall reliability of bash execution across a session.

Why This Matters

Agent Arena’s signal leaderboard is used by engineering teams to evaluate which models are safe to deploy in automated pipelines. A metric that penalises correct behaviour or ignores self-correction inflates failure rates for capable agents and deflates them for agents that produce quiet errors in piped workflows.

The switch to LLM-as-judge is consistent with how the wider agent evaluation space is evolving. Rule-based detection works cleanly for simple pass/fail assertions but breaks down when agent behaviour is conditional, contextual, or self-healing. LiveBench and similar platforms have made similar moves, using model judges for tasks that resist clean programmatic evaluation.

Arena.ai said the change results in a “more accurate assessment of whether a bash call failed.” Rankings on the Bash Reliability signal are expected to shift from prior Bash Recovery rankings as the new scores propagate.