Exploring Large Language Model Agents in Cybersecurity: A Literature Review with Experiments

Billingsley, Brett, Fassett, Jayden, Werth, Aaron et al. · 2025

Read Exploring Large Language Model Agents in Cybersecurity: A Literature Review with Experiments by Billingsley, Brett, Fassett, Jayden, Werth, Aaron et al. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/doe-osti-3009426-exploring-large-language-model-agents-in.txt.

Introduction

Oak Ridge National Laboratory report ORNL/TM-2025/4288, by Brett Billingsley, Jayden Fassett, Aaron Werth, and Mathew Singleton, dated December 2025, reviews large language model agents in cybersecurity and adds the authors' own experiments. The question is how far an AI agent can go in offensive and defensive security work.

Systems surveyed include PentestGPT, which works as an interactive assistant: it reasons, summarizes, and recommends commands while a human executes them, keeping a task tree for context. The review notes PentestGPT is now largely deprecated in favour of a successor framework, Cybersecurity AI. Readers get a map of a fast-moving tool landscape with honest limits attached.

Catalog record

CREATORBillingsley, Brett, Fassett, Jayden, Werth, Aaron et al.
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
LICENSEPublic Domain
RIGHTSPUBLIC DOMAIN
SOURCEDOE OSTI

Source: https://www.osti.gov/biblio/3009426

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ORNL/TM-2025/4288 Exploring Large Language Model Agents in Cybersecurity: A Literature Review with Experiments Brett Billingsley Jayden Fassett Aaron Werth Approved for public release. Mathew Singleton Distribution is unlimited. December 2025 ORNL IS MANAGED BY UT-BATTELLE LLC FOR THE US DEPARTMENT OF ENERGY DOCUMENT AVAILABILITY Online Access: US Department of Energy (DOE) reports produced after 1991 and a growing number of pre-1991 documents are available free via https://www.osti.gov/. The public may also search the National Technical Information Service’s National Technical Re- ports Library (NTRL) for reports not available in digital format. DOE and DOE contractors should contact DOE’s Office of Scientific and Technical Information (OSTI) for reports not currently available in digital format: US Department of Energy Office of Scientific and Technical Information PO Box 62 Oak Ridge, TN 37831-0062 Telephone: (865) 576-8401 Fax: (865) 576-5728 Email: [email protected] Website: https://www.osti.gov/ This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal lia- bility or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or rep- resents that its use would not infringe privately owned rights. Refer- ence herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not nec- essarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not neces- sarily state or reflect those of the United States Government or any agency thereof. ORNL/TM-2025/4288 Cyber Resilience and Intelligence Division EXPLORING LARGE LANGUAGE MODEL AGENTS IN CYBERSECURITY: A LITERATURE REVIEW WITH EXPERIMENTS Brett Billingsley Jayden Fassett Aaron Werth Mathew Singleton December 2025 Prepared by OAK RIDGE NATIONAL LABORATORY Oak Ridge, TN 37831 managed by UT-BATTELLE LLC for the US DEPARTMENT OF ENERGY under contract DE-AC05-00OR22725 CONTENTS LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2. BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1 LARGE LANGUAGE MODELS (LLMs) . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.2 RETRIEVAL AUGMENTED GENERATION . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.3 PENETRATION TESTING PROCESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. LITERATURE REVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3.1 LLM-DIRECTED AGENT MODELS IN CYBERSPACE . . . . . . . . . . . . . . . . . . 5 3.1.1 Hierarchical Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3.1.2 Context Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.2 PENHEAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.2.1 PenHeal Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.3 PENTESTGPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.4 CIPHER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.5 HACKSYNTH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.5.1 HackSynth Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.5.2 Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.6 CYBERSECURITY AI (CAI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.6.1 CAI Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.6.2 Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . .

Read the complete text