// HACKER NEWS — CYBERSECURITY
When LLM judges agree, should we believe them?
Imagine evaluating a retrieval-augmented-generation system. A user asks a question, the system retrieves a text passage, and an LLM judge decides whether it’s relevant. To reduce noise, you ask several judge models to evaluate the same passage. Eight say “relevant”; two say “not relevant”.
Eight out of 10 feels convincing. But the important question is not only how many judges agreed but how independently they arrived at that agreement.
If the eight agreeing judges are genuinely different sources of evidence, then agreement is a strong signal. But if they share a prompt template, a training lineage, a model family, or a common blind spot, they may be repeating the same mistake. The vote count makes the evidence look stronger than it really is.
Our paper “Dependence-aware label aggregation for LLM-as-a-judge via Ising models,” coauthored with Shiva Kasiviswanathan and presented at this year’s International Conference on Machine Learning (ICML), addresses this problem. We present a method for assessing the correlations between judges’ outputs and adjusting the aggregate score accordingly, to ensure a diversity of opinion.
In tests on three different tasks, our method outperformed the best-performing baseline — a panel of judges weighted according to historical accuracy — by 9% to 14% on standard metrics.
The attraction of majority vote is its simplicity. Every judge gets one vote, and the answer with more votes wins. Weighted majority vote is a natural improvement: judges that appear more accurate get more influence.
Both approaches are useful baselines. But they are built around the same simplified view of the judge panel: judges that get the wrong answer are treated as though they make their errors independently.
That assumption is often too optimistic for LLM-as-a-judge systems. Two judges may fail together because they interpret the rubric similarly. Several judges may be prompted with the same examples and therefore inherit the same evaluation bias. A group of related models may be sensitive to the same phrasing. In these cases, a majority can be less informative than it appears.
A better aggregator would treat the panel as a network of judges. Each judge still has its own reliability profile, but pairs of judges can also have relationships. Some pairs agree more often than their individual reliability profiles would predict, including on shared mistakes. Other pairs provide more complementary perspectives.
We model these relationships with an Ising model, a statistical model that can represent pairwise dependence between binary variables. In the LLM-as-a-judge context, the aggregator learns both judge skill and judge similarity.