Towards Automated Pentesting with Large Language Models
Ricardo Bessa, Rui Claro, João Trindade, João M. Lourenço · 2026
Read Towards Automated Pentesting with Large Language Models by Ricardo Bessa, Rui Claro, João Trindade, João M. Lourenço (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11772.txt.
Introduction
Bessa and three colleagues present RedShell, a private lean framework using fine-tuned LLMs to help testers write offensive PowerShell code. While attackers use dark LLMs for malicious code, ethical hackers can automate old pentest workflows the same way. RedShell trained on a malicious PowerShell dataset, enhanced with hand-picked samples. Tests show over 90 percent valid syntax and strong meaning alignment with reference snippets, beating counterparts on edit distance above 50 percent code similarity. Functional tests stress execution reliability in real-world tests.
Catalog record
| CREATOR | Ricardo Bessa, Rui Claro, João Trindade, João M. Lourenço |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49428 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11772
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Automated Pentesting with Large Language Models Ricardo Bessa, Rui Claro, João Trindade, João M. Lourenço Abstract Large Language Models (LLMs) are redefining offensive cybersecurity by allowing the generation of harmful machine code with minimal human intervention. While attackers take advantage of dark LLMs such as XXXGPT and WolfGPT to produce malicious code, ethical hackers can follow similar approaches to automate traditional pentesting workflows. In this work, we present RedShell, a privacy-preserving, hardware-efficient framework that leverages fine-tuned LLMs to assist pentesters in generating offensive PowerShell code targeting Microsoft Windows vulnerabilities. RedShell was trained on a malicious PowerShell dataset from the literature, which we further enhanced with manually curated code samples. Experiments show that our framework achieves over 90% syntactic validity in generated samples and strong semantic alignment with reference pentesting snippets, outperforming state-of-the-art counterparts in distance metrics such as edit distance (above 50% average code similarity). Additionally, functional experiments emphasize the execution reliability of the snippets produced by RedShell in a testing scenario that mirrors real-world settings. This work sheds light on the state-of-the-art research in the field of Generative AI applied to malicious code generation and automated testing, acknowledging the potential benefits that LLMs hold within controlled environments such as pentesting. 1 Introduction In recent years, LLMs sparked a revolution in natural language processing through advanced transformers [ 52 ] . Models such as ChatGPT [ 43 ] and DeepSeek [ 10 ] have demonstrated strong capabilities while performing generic tasks, including text classification and language understanding [ 55 ] . In addition, fine-tuning techniques have been employed to adapt LLMs to address more specific challenges, with Generative AI becoming pervasive in many software-engineering workflows, particularly in automated testing [ 17 ] . In the offensive cybersecurity field, AI models such as XXXGPT and WolfGPT arise as a new tool for malicious actors to generate harmful code [ 45 ] . However, ethical hackers can also take advantage of LLMs to develop malicious code generators, providing more automation to traditional pentest audits. Penetration testing, often referred to as pentesting, is a crucial activity for red teams, which are responsible for testing the cybersecurity effectiveness of a system through simulated cyber-attacks [ 53 ] . By detecting potential security lapses in a target system, pentesters are able to prevent attacks that could cause harm to that system and its users. Ethical hackers often rely on Assembly shellcode, PowerShell scripts, and other forms of malicious code to identify and exploit security vulnerabilities. However, regardless of the exploitation technique being employed, writing offensive code demands significant time, effort, and expertise from pentesters [ 13 ] . While LLMs offer a promising path towards automated malicious code generation, this approach also remains challenging due to the limited public availability of offensive training data. In this paper, we present RedShell, a framework that incorporates locally fine-tuned LLMs to support malicious PowerShell code generation, providing more automation to pentesting activities targeting Microsoft Windows. To build RedShell, we followed a privacy-preserving and hardware-efficient methodology that can be generalized to support the production of other forms of offensive code beyond PowerShell. We also build on a malicious PowerShell dataset from the literature, extending it with code samples from various cybersecurity frameworks, to train and evaluate models in offensive PowerShell code generation. The evaluation focuses on assessing the quality of the generated snippets by examining their syntactic, semantic, and functional correctness, as well as performing a comparative analysis of the generation capabilities of different LLMs. Experiments show that our framework was able to generate syntactically valid PowerShell code, with over 90% of the generated samples successfully parsed without errors. Additionally, RedShell produced samples closely aligned with reference snippets, achieving more than 50% average code similarity across standard metrics such as edit distance. These results position RedShell as a competitive solution for automated malicious PowerShell generation, outperforming the output semantics of reference models in the cybersecurity domain. Furthermore, functional experiments conducted within a controlled environment demonstrate the execution reliability of the snippets produced by RedShell in a realistic pentesting scenario. The remainder of this paper is organized as follows: Section 2 reviews background concepts and related work; Section 3 introduces an extended version of a malicious PowerShell dataset from the literature; Section 4 details the design of RedShell; Section 5 presents experiments validating RedShell’s generation capabilities; Section 6 discusses ethical considerations regarding malicious AI usage, and finally, Section 7 concludes the paper and describes future directions. 2 Background & Related Work LLMs registered a high popularity in recent years, with the AI-community actively searching for new approaches to develop generative models that deliver strong performance on both general and domain-specific tasks. In addition, engineers are taking advantage of these novel technologies to automate complex and time-consuming processes, allowing human operators to focus on more critical aspects of their workflows. We now discuss core concepts within the fields of pentesting and Generative AI, and then we review related work on the application of LLMs in cybersecurity. 2.1 Traditional Pentesting Pentesting. While conducting pentest audits, ethical hackers follow an offensive methodology that starts from analyzing a target network or system, and culminates in the emulation of real adversary behaviors [ 53 ] . Through pentesting, red teams can assess the functional aspects of a system and detect potential security vulnerabilities that could be exploited by real attackers. Weidman [ 54 ] highlights the differences between pentesting and vulnerability assessment. In a vulnerability assessment, security professionals identify core system vulnerabilities that could be exploited by malicious actors. Pentesting extends vulnerability assessment by further performing the exploitation of the detected flaws to assess what hackers might achieve after a successful attack. MITRE ATT&CK. The MITRE ATT&CK framework [ 9 ] provides a comprehensive taxonomy of the tactics, techniques, and procedures (TTPs) employed by real adversaries to conduct offensive operations. The framework classifies malicious activities into 14 tactical groups representing different stages of an attack. Built on real-world observations, MITRE ATT&CK offers a valuable knowledge base for security professionals to develop threat models in both the private sector and the cybersecurity product and service community [ 47 ] . The framework covers the adversarial behaviors that can be performed against different operating systems, including detailed information describing the most common tools and methodologies to carry out those attacks. PowerShell. As Microsoft Windows remains one of the most frequently targeted operating systems, PowerShell has evolved into an effective exploitation framework for pentesters [ 31 ] . The PowerShell programming language is included by default in modern versions of Windows, providing a flexible scripting environment for automating administrative tasks and managing remote services. PowerShell’s versatility also makes it an attractive vector for adversaries, who can leverage its built-in functionality to execute custom payloads without downloading additional malicious tools on the target system. In 2024, PowerShell ranked fourth among the most frequently observed MITRE ATT&CK techniques in confirmed threats, affecting 20.4% of customers and accounting for 684 detected incidents across the Red Canary customer base [ 5 ] . Reported occurrences showcase malicious agents actively abusing PowerShell to obfuscate offensive activities, gather information, change system configurations, and remotely download and execute arbitrary code and binaries such as Mimikatz [ 11 ] (to dump in-memory secrets), and AzureAD