ProbeLLM : Automating Principled Diagnosis of LLM Failures

Yue Huang, Zhengzhe Jiang, Yuchen Ma, Yu Jiang, Xiangqi Wang, Yujun Zhou, Yuexing Hao, Kehan Guo, Pin-Yu Chen, Stefan Feuerriegel, Xiangliang Zhang · 2026

Read ProbeLLM : Automating Principled Diagnosis of LLM Failures by Yue Huang, Zhengzhe Jiang, Yuchen Ma, Yu Jiang, Xiangqi Wang, Yujun Zhou, Yuexing Hao, Kehan Guo, Pin-Yu Chen, Stefan Feuerriegel, Xiangliang Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12966.txt.

Description

Static benchmarks age fast as language models evolve, and isolated failure examples say little about systematic weakness. Yue Huang and ten co-authors propose ProbeLLM, a benchmark-agnostic framework that turns failure discovery into structured failure modes. Probing becomes hierarchical Monte Carlo tree search, splitting a limited budget between exploring new failure regions and refining recurring error patterns.

Only verifiable test cases are used, with tool-augmented generation and verification grounding discoveries in reliable evidence. Found failures are then grouped into interpretable modes using embeddings tuned to failures and induction that respects mode boundaries. Across benchmarks and models, ProbeLLM reveals broader, cleaner, and finer-grained weakness landscapes than static tests or earlier automated methods, supporting a shift from case counting to principled weakness discovery.

Catalog record

CREATORYue Huang, Zhengzhe Jiang, Yuchen Ma, Yu Jiang, Xiangqi Wang, Yujun Zhou, Yuexing Hao, Kehan Guo, Pin-Yu Chen, Stefan Feuerriegel, Xiangliang Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-12966

Browse the full library · About the Tower of Babel