JFTA-Bench: Evaluate LLM’s Ability of Tracking and Analyzing Malfunctions Using Fault Trees
Yuhui Wang, Zhixiong Yang, Ming Zhang, Shihan Dou, Zhiheng Xi, Enyu Zhou, Senjie Jin, Yujiong Shen, Dingwei Zhu, Yi Dong, Tao Gui, Qi Zhang, Xuanjing Huang · 2026
Read JFTA-Bench: Evaluate LLM’s Ability of Tracking and Analyzing Malfunctions Using Fault Trees by Yuhui Wang, Zhixiong Yang, Ming Zhang, Shihan Dou, Zhiheng Xi, Enyu Zhou, Senjie Jin, Yujiong Shen, Dingwei Zhu, Yi Dong, Tao Gui, Qi Zhang, Xuanjing Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22978.txt.
Description
Wang and co-authors present JFTA-Bench, a benchmark for language models that reason with fault trees. Fault trees guide maintenance teams to locate system failures. Their text format lets models read fault trees stored as images. The benchmark uses multi-turn dialogue with thousands of entries. Long dialogues test task tracking, and built-in rollback steps mimic user errors. The team also trains a model that plays the vague user. Gemini 2.5 Pro scores best among the models tested.
Catalog record
| CREATOR | Yuhui Wang, Zhixiong Yang, Ming Zhang, Shihan Dou, Zhiheng Xi, Enyu Zhou, Senjie Jin, Yujiong Shen, Dingwei Zhu, Yi Dong, Tao Gui, Qi Zhang, Xuanjing Huang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60827 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22978