ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, Anand Mudgerikar · 2025

Read ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation by Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, Anand Mudgerikar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14201.txt.

Description

Security analysts chasing a cyber threat must sift through huge piles of logs and follow chains of evidence across many steps. Yiran Wu and co-authors build ExCyTIn-Bench, the first test set for judging how well language-model agents handle that kind of investigation. The benchmark comes from a controlled Microsoft Azure setup with a database of 57 log tables drawn from Sentinel and related services.

Questions are generated from threat investigation graphs built with expert detection rules. Each question pairs a starting node, which gives background, with an end node that holds the answer, so grading is automatic and explainable. Across 7,542 questions, the best model tested earns a reward of only 0.606, leaving much room to improve.

Catalog record

CREATORYiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, Anand Mudgerikar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE126 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-14201

Browse the full library · About the Tower of Babel