← Back to feed

When eight of ten LLM judges agree, the instinct is to trust the majority. Amazon Science researchers presented at ICML 2026 argue that instinct is wrong — and have built a better system to replace it.

The paper, “Dependence-aware label aggregation for LLM-as-a-judge via Ising models,” co-authored by Krishna Balasubramanian, Sasha Podkopaev, and Shiva Kasiviswanathan, targets a structural weakness in multi-judge evaluation panels that most practitioners assume away.

The Correlation Problem

Standard majority-vote and weighted-majority-vote approaches treat each judge’s error as independent. That assumption breaks down when judges share a prompt template, a training lineage, a model family, or a common blind spot. Under those conditions, a group of models can all make the same mistake, and the vote count inflates confidence in a wrong answer.

The practical example from the paper: evaluating a retrieval-augmented-generation system where an LLM judge decides whether a retrieved passage is relevant. Eight judges say “relevant”; two say “not relevant.” The eight agreeing judges may simply share the same evaluation bias — the apparent consensus carries no more information than a single biased judge.

The Ising Model Fix

The Amazon team models the judge panel as a network of pairwise dependencies rather than a pool of independent voters. Each judge retains its own reliability profile, but pairs of judges also have relationships — some agree more than their individual accuracy would predict, including on shared errors.

Ising models, borrowed from statistical physics where they describe correlated binary spin states, handle this naturally. The aggregator learns both per-judge skill and pairwise judge similarity simultaneously. When two judges are highly correlated, their agreement adds less information to the aggregate than if they were independent.

Results

Tested across three evaluation tasks, the dependence-aware aggregator outperformed the best-performing baseline — a weighted majority vote calibrated to historical accuracy — by 9% to 14% on standard metrics.

Why This Matters

LLM-as-judge is now load-bearing infrastructure. It drives preference data for RLHF, powers automated red-teaming, evaluates RAG pipelines, and rates outputs in production monitoring systems. Most of those deployments use some version of majority vote across a panel of models.

The implicit assumption is that more judges means more signal. The Amazon paper shows the relationship is nonlinear: if judges are correlated, adding more of the same family improves precision only marginally. Diversity of judge lineage matters as much as count.

Key Numbers

• Accuracy gain over best baseline: 9-14% across three tasks
• Baseline compared: weighted majority vote (judges weighted by historical accuracy)
• Method: Ising-model pairwise dependence estimation, applied to multi-judge label aggregation
• Venue: ICML 2026