Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks

Lining Hu, Ting Liu, Yuzhuo Fu · 2026

Read Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks by Lining Hu, Ting Liu, Yuzhuo Fu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29159.txt.

Description

Hu, Liu and Fu audit how offline root-cause analysis benchmarks rank competing methods. Such benchmarks, used to judge systems that diagnose faults in software services, commonly pool scores across many subsystems into one leaderboard. Engineers then read the pooled winner as the best choice for their own subsystem, an inference the authors question.

Their audit covers three public benchmark families, eleven subsystems and 778 matched scoring units, with four methods compared on identical cases. All six pairwise comparisons showed subsystem-level effects in both directions: each method won somewhere and lost elsewhere. Every pooled prediction interval crossed zero, and interaction tests rejected the idea that cases are interchangeable in most pairs. Selecting on all but one subsystem often picked the loser for the held-out one, with regret reaching about twenty-five points in the worst case. The team release a compact audit module recomputing these stability checks.

Catalog record

CREATORLining Hu, Ting Liu, Yuzhuo Fu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29159

Browse the full library · About the Tower of Babel