ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation
Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, Anand Mudgerikar · 2025
Read ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation by Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, Anand Mudgerikar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14201.txt.
Description
Security analysts chasing a cyber threat must sift through huge piles of logs and follow chains of evidence across many steps. Yiran Wu and co-authors build ExCyTIn-Bench, the first test set for judging how well language-model agents handle that kind of investigation. The benchmark comes from a controlled Microsoft Azure setup with a database of 57 log tables drawn from Sentinel and related services.
Questions are generated from threat investigation graphs built with expert detection rules. Each question pairs a starting node, which gives background, with an end node that holds the answer, so grading is automatic and explainable. Across 7,542 questions, the best model tested earns a reward of only 0.606, leaving much room to improve.
Catalog record
| CREATOR | Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, Anand Mudgerikar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 126 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-14201