Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hunting
Xiaoqun Liu, Feiyang Yu, Xi Li, Guanhua Yan, Ping Yang, Zhaohan Xi · 2025
Read Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hunting by Xiaoqun Liu, Feiyang Yu, Xi Li, Guanhua Yan, Ping Yang, Zhaohan Xi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11901.txt.
Introduction
Liu and co-authors build CYBERTEAM, a benchmark for testing whether large language models can help defenders hunt cyber threats. Blue teams trace attacks step by step, from working out who is behind an incident to responding to it, and each step depends on the ones before.
The benchmark turns that workflow into thirty tasks carried out through nine embodied functions, so a model must act in sequence rather than just answer questions. The authors test leading language models and security agents, comparing guided function-calling with plain prompting. The results map where today's models genuinely help threat hunters and where they still fall short, giving builders a firmer base for real security tools.
Catalog record
| CREATOR | Xiaoqun Liu, Feiyang Yu, Xi Li, Guanhua Yan, Ping Yang, Zhaohan Xi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11901
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hunting Xiaoqun Liu, Feiyang Yu, Xi Li, Guanhua Yan, Ping Yang, Zhaohan Xi Abstract As cyber threats continue to grow in scale and sophistication, blue team defenders increasingly require advanced tools to proactively detect and mitigate risks. Large Language Models (LLMs) offer promising capabilities for enhancing threat analysis. However, their effectiveness in real-world blue team threat-hunting scenarios remains insufficiently explored. In this paper, we present CyberTeam , a benchmark designed to guide LLMs in blue teaming practice. CyberTeam constructs an embodied environment in two stages. First, it models realistic threat-hunting workflows by capturing the dependencies among analytical tasks from threat attribution to incident response. Next, each task is addressed through a set of embodied functions tailored to its specific analytical requirements. This transforms the overall threat-hunting process into a structured sequence of function-driven operations, where each node represents a discrete function and edges define the execution order. Guided by this framework, LLMs are directed to perform threat-hunting tasks through modular steps. Overall, CyberTeam integrates 30 tasks and 9 embodied functions, guiding LLMs through pipelined threat analysis. We evaluate leading LLMs and state-of-the-art cybersecurity agents, comparing CyberTeam ’s embodied function-calling against fundamental elicitation strategies. Our results offer valuable insights into the current capabilities and limitations of LLMs in threat hunting, laying the foundation for the practical adoption in real-world cybersecurity applications. 1 Introduction The increasing frequency and sophistication of cyber threats continue to pose significant challenges to organizational security. In 2024 alone, over 11,000 more ( 38\% increase!) vulnerabilities were reported compared to 2023, as evidenced by the MITRE CVE database mitre_cve . Defenders, commonly known as the blue team diogenes2018cybersecurity ; rajendran2011blue , are under increasing pressure to identify, analyze, and respond to malicious activities in a timely and accurate manner, a process termed as threat hunting . Traditionally, threat hunting has been a labor-intensive process, relying heavily on the expertise of analysts to sift through logs, correlate indicators of compromise (IOCs), and construct hypotheses about potential attacks dash2022review ; gioti2024advancements . This process demands both deep domain knowledge and an ability to integrate fragmented evidence from multiple sources under time constraints alshumrani2023unified ; yu2010vqsvm ; zhou2017knowledge . Recent advances in Large Language Models (LLMs) have demonstrated impressive potential to augment cybersecurity practices, including malware analysis abusitta2021malware ; al2024exploring ; qian2025lamd ; devadiga2023gleam , penetration testing deng2023pentestgpt ; deng2024pentestgpt ; happe2023getting ; muzsai2024hacksynth , and fuzzing zhang2025llms ; oliinyk2024fuzzing ; black2024evaluating . Building on this progress, there is growing interest in leveraging LLMs to automate or assist in threat hunting, enabling blue team defenders to scale their investigations across complex threat landscapes and respond to incidents more effectively. However, despite this momentum, the application of LLMs in blue team threat hunting remains underdeveloped. Existing frameworks tend to focus on isolated analytical tasks sehgal2023cybersecurity ; faghihi2023gluecons ; dash2022review ; borghesi2020improving ; groza2015capturing , such as generating advisory recommendations without integrating earlier steps like threat group attribution. This fragmented design limits our understanding of how LLMs perform within complex, interdependent threat-hunting workflows. To address this gap, we introduce CyberTeam , a practical benchmark designed to rigorously evaluate and guide the use of LLMs in blue team threat hunting. CyberTeam supports blue-team threat-hunting workflows through the following aspects: Figure 1: A threat hunting example equipped with the embodied functions. Function names: NER–named entity recognition, REX–regex parsing, MAP–text mapping, RAG–retrieval-augmented generation, CLS–classification, SUM–summarization. Broader Coverage. CyberTeam is constructed from a diverse and large-scale repository of threat intelligence data sourced from 23 vulnerability databases, including MITRE series mitre_attack , NVD nvd , Exploit-DB exploitdb , VulDB vuldb , and CISE cise , as well as reporting platforms such as Red Hat Bugzilla bugzilla , Oracle Security Alerts oracle_security_alerts , and IBM X-Force ibm_xforce . This broad and representative collection is essential for capturing the complexity and variability of the modern threat landscape, enabling realistic support for both threat investigation and incident response. In addition, CyberTeam includes a significantly larger number of tasks and samples than existing cybersecurity benchmarks jimenez2023swe ; li2025iris ; alam2024ctibench ; ji2024sevenllm , as summarized in Table 1 . This extensive coverage allows for a more comprehensive and nuanced evaluation of LLM performance across a wide range of threat-hunting scenarios. Embodied Environment. An important feature of CyberTeam is its structured, modular workflow for guiding LLMs within an embodied environment yang2024holodeck ; cheng2025embodiedeval . This design is inspired by blue team practices, where analysts typically follow standardized procedures to interpret threat metadata and conduct investigations sehgal2023cybersecurity ; diogenes2018cybersecurity ; brotherston2024defensive . However, strict adherence to such procedures can limit adaptability when analyzing unstructured threat logs or addressing emerging, zero-day threats. To balance standardization and adaptability, CyberTeam integrates a set of embodied functions that regulate LLM behavior while allowing for open-ended reasoning where needed. As illustrated in Figure 1 , CyberTeam first models the dependency structure among threat-hunting objectives (e.g., attribution, behavior analysis, mitigation) as a task chain, and then maps this chain into a corresponding sequence of embodied functions. In this process, functions such as NER enforce structured outputs (e.g., extracting threat actor entities), while functions like RAG support more flexible reasoning (e.g., summarizing relevant patching strategies). This design enables CyberTeam to constrain model behavior when precision is essential, while still allowing LLMs to engage in context-sensitive generation when analytical creativity or abstraction is required. Evaluation Strategy. CyberTeam incorporates agent-based evaluation strategies tailored to each threat-hunting objective. We benchmark leading LLMs and state-of-the-art (SOTA) cybersecurity agents, comparing CyberTeam ’s embodied function-calling approach with popular elicitation strategies such as In-Context Learning (ICL) dong2022survey , Chain-of-Thought (CoT) wei2022chain , Tree-of-Thought (ToT) yao2023tree . Our evaluation provides insights into the capabilities and limitations of current models across 30 tasks. We further analyze LLMs’ capabilities in automatically identifying task dependencies, selecting appropriate embodied functions, and handling noisy threat log inputs. These experiments reveal not only underlying model limitations, such as reasoning errors and sensitivity to incomplete or ambiguous input, but also provide practical insights for blue team analysts aiming to integrate LLMs into real-world cyber defense workflows. In summary, this paper makes the following contributions: (1) We introduce CyberTeam , a practice-informed, broadly scoped benchmark that enables rigorous evaluation of LLMs for blue team threat hunting, (2) we construct an embodied environment that models the dependencies among threat-hunting tasks and guides LLMs through standardized yet flexible reasoning workflow, (3) we conduct comprehensive evaluations and provide insights to improve LLM performance among threat-hunting scenarios. Table 1: Comparison of cybersecurity benchmarks for LLMs. Benchmark Focus #Data #Task #Source Coverage Unique Feature CWE-Bench-Java li2025iris Java vulnerability 120 4 1 Four CWE classes Large-scale Java codes CTIBench alam2024ctibench Cyber Threat Intelligence 2,500 3 6 CVE, CWE, CVSS, ATT&CK Multi-choice questions (MCQ) SevenLLM-Bench ji2024sevenllm Report understanding 91,401 28 N/A Bilingual instruction corpus Synthetic Data, MCQ, QA SWE-Bench jimenez2023swe Software bug fixing 2,294 12 1 GitHub issues Python repository CyberTeam (Ours) Blue-team threat hunting 452,293 30 23 Threat-hunting lifecycle ( 3.1 ) Open Generation, Embodied Env 2 Related Work LLMs for Cybersecurity. Recently, LLMs have shown promise in enhancing cybersecurity tasks such as malware classification abusitta2021malware ; al2024exploring ; qian2025lamd ; devadiga2023gleam ; liu2025cyber , code vulnerability detection russell2018automated ; lu2024grace ; sheng2024lprotector