CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning
Lauren Deason*†, Adam Bali†, Ciprian Bejean‡, Diana Bolocan*‡, James Crnkovich*†, Ioana Croitoru*‡, Krishna Durai†, Chase Midler‡, Calin Miron‡, David Molnar*†, Brad Moon‡, Bruno Ostarcevic‡, Alberto Peltea‡, Matt Rosenberg‡, Catalin Sandu‡, Arthur Saputkin†, Sagar Shah*†, Daniel Stan‡, Ernest Szocs · 2025
Read CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning by Lauren Deason*†, Adam Bali†, Ciprian Bejean‡, Diana Bolocan*‡, James Crnkovich*†, Ioana Croitoru*‡, Krishna Durai†, Chase Midler‡, Calin Miron‡, David Molnar*†, Brad Moon‡, Bruno Ostarcevic‡, Alberto Peltea‡, Matt Rosenberg‡, Catalin Sandu‡, Arthur Saputkin†, Sagar Shah*†, Daniel Stan‡, Ernest Szocs (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20166.txt.
Description
Lauren Deason and twenty-two co-authors present a new benchmark suite in this 2025 arXiv paper on cyber defense. Security operations centers face more alerts and threat reports than staff can handle. Large language models could help, but existing tests do not measure the tasks defenders actually do. CyberSOCEval was built to close that gap.
The suite covers two core tasks: malware analysis and threat intelligence reasoning. It forms part of the open CyberSecEval 4 release. Larger and newer models scored better overall, matching the usual scaling pattern. Reasoning models, though, gained less from extra test-time thinking than they do in coding or mathematics. No model came close to topping the tests. The authors offer the suite as an open target for building stronger defensive tools.
Catalog record
| CREATOR | Lauren Deason*†, Adam Bali†, Ciprian Bejean‡, Diana Bolocan*‡, James Crnkovich*†, Ioana Croitoru*‡, Krishna Durai†, Chase Midler‡, Calin Miron‡, David Molnar*†, Brad Moon‡, Bruno Ostarcevic‡, Alberto Peltea‡, Matt Rosenberg‡, Catalin Sandu‡, Arthur Saputkin†, Sagar Shah*†, Daniel Stan‡, Ernest Szocs |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20166