Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks
Lining Hu, Ting Liu, Yuzhuo Fu · 2026
Read Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks by Lining Hu, Ting Liu, Yuzhuo Fu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29159.txt.
Description
Hu, Liu and Fu audit how offline root-cause analysis benchmarks rank competing methods. Such benchmarks, used to judge systems that diagnose faults in software services, commonly pool scores across many subsystems into one leaderboard. Engineers then read the pooled winner as the best choice for their own subsystem, an inference the authors question.
Their audit covers three public benchmark families, eleven subsystems and 778 matched scoring units, with four methods compared on identical cases. All six pairwise comparisons showed subsystem-level effects in both directions: each method won somewhere and lost elsewhere. Every pooled prediction interval crossed zero, and interaction tests rejected the idea that cases are interchangeable in most pairs. Selecting on all but one subsystem often picked the loser for the held-out one, with regret reaching about twenty-five points in the worst case. The team release a compact audit module recomputing these stability checks.
Catalog record
| CREATOR | Lining Hu, Ting Liu, Yuzhuo Fu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29159