Cochise: A Reference Harness for Autonomous Penetration Testing DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Security and privacy Software and application security

Andreas Happe, Jürgen Cito · 2026

Read Cochise: A Reference Harness for Autonomous Penetration Testing DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Security and privacy Software and application security by Andreas Happe, Jürgen Cito (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11671.txt.

Introduction

Happe and Cito present Cochise, a reference harness for penetration testing. LLM penetration testing shows promise, but existing systems bundle architecture and tool choices. Cochise is a 630-line Python tool connecting over SSH to a Linux host to attack targets from that jump host. It uses a Planner-Executor design: the planner holds state while an executor issues SSH commands and self-corrects from outputs. Tested on the Game of Active Directory testbed, Cochise ships replay and analysis tools plus logs for studying agents without the heavy testbed.

Catalog record

CREATORAndreas Happe, Jürgen Cito
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18807
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cochise: A Reference Harness for Autonomous Penetration Testing DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Security and privacy Software and application security Andreas Happe, Jürgen Cito Abstract . Recent work on LLM-driven autonomous penetration testing reports promising results, but existing systems often combine many architectural, prompting, and tool-integration choices, making it difficult to tell what is gained over a simple agent scaffold. We present cochise, a 597 LOC Python reference harness for autonomous penetration-testing experiments. Cochise connects an LLM-driven agent to a Linux execution host over SSH and supports controlled target environments reachable from that jump host. The prototype implements a separated Planner–Executor architecture in which long-term state is maintained outside the LLM context, while a ReAct-style executor issues commands over SSH and self-corrects based on command outputs. The scenario prompt can be adapted to different target environments. To demonstrate the efficacy of our minimal harness, we evaluate it against a live third-party testbed called Game of Active Directory (GOAD). Alongside the harness, we release replay and analysis tools: (i) cochise-replay for offline visualization of captured runs, (ii) cochise-analyze-logs and cochise-analyze-graphs for cost, token, duration, and compromise analysis, and (iii) a corpus of JSON trajectory logs from GOAD runs, allowing researchers to study agent behavior without provisioning the 48–-64 GB RAM / 190 GB storage testbed themselves. Cochise is intended not as a state-of-the-art pen-testing agent, but as reusable experimental infrastructure for comparing models, agent architectures, and penetration-testing traces. Tool demo video available at https://youtu.be/2mQimB1ufyI . 1. Introduction There has been large academic and industrial interest in using LLMs for autonomous network penetration testing. While publications provide performance numbers for their respective prototypes, reusable execution and logging infrastructure is typically missing. This makes it difficult to compare models and architectures under a common protocol or to reuse traces and analysis pipelines across studies. Cochise is intended as a reference harness 1 1 1 A small, stable scaffold/harness against which alternative agent designs can be ablated. rather than as a high-performance pentesting agent: it fixes the execution interface, model abstraction, state handling, and trajectory format so that researchers can compare models, architectural variants, and agent traces under a common protocol. By using the LiteLLM framework, it allows easy exchange of the utilized LLM; its configurable scenario prompt allows integration into diverse use-cases. Its implementation as readable and minimal code base with 597 lines of python code allows both usage as an independent reference harness as well as starting point for new prototypes. It provides a thorough JSON based trajectory log format and includes tools for subsequent data analysis of those logs. Provided Artifacts On the tooling side, we provide the cochise autonomous penetration-testing tool. A separate tool, cochise-replay , allows offline replay of captured log files for later visualization. For log analysis we provide cochise-analyze-logs to analyze collections of generated trajectory logs with regard to used models, token utilization, costs, and success-rates. To aid researchers, it allows export as L a T e X tables while cochise-analyze-graphs allows to generate graphs. We also provide a collection of JSON - based penetration-testing trajectory logs to enable research without incurring token costs. All tools as well as a data-set of example penetration-testing trajectories are shared as a public github repository at https://github.com/andreashappe/cochise under a permissive open-source license. All released cochise versions are automatically archived on zenodo, the paper version is https://zenodo.org/records/20126672 . Intended Users The primary audience is researchers who need a controllable harness for studying autonomous penetration-testing agents. Researchers can reuse Cochise’s SSH execution interface, trajectory format, replay tooling, and analysis scripts; compare LLMs through the LiteLLM backend; or analyze the released traces without running the live testbed. 2. Motivation, Background And Related Work There has been high academic interest in using LLMs for autonomous penetration testing, with current prototypes including incalmo ( Singer et al. 2025 ) , cAI ( Mayoral-Vilches et al. 2025 ) , and pentestGPT 2 2 2 We include pentestGPT although it is single-host and not multi-host centric. ( Deng et al. 2026 ) . These prototypes commonly focus on increasing their respective success rates, provide multiple operation modes, and are subject to constant tinkering. A prior version of our research ( Happe and Cito 2025 ) was used by PentestGPT as a point of comparison, indicating demand for reusable (baseline) scaffolds in this space. This inspired us to write cochise , a documented reference/baseline harness with explicit execution, logging, replay, and analysis interfaces. Penetration Testing as an SE Problem While our target domain is security-centric, the problem an autonomous penetration-testing agent must solve originates in the software engineering domain. Realistic security evaluation testbeds are highly dynamic stateful environments where actions can trigger destructive side-effects. Any interaction of the agent with the environment may produce adverse effects, e.g., invoking an exploit can crash the target system, enumeration actions can trigger intrusion detection systems, or running password-based attacks can result in account lock-outs. In addition, defenses such as automated EDR solutions act as active adversaries, creating attacker/defender setups. To achieve success, agents must cope with these side-effects and must combine findings into successful attack paths, making our trajectories inherently multi-step. As the agent can run arbitrary commands on the execution host (Section 3.1 ), the action space is unbound. In addition, penetration-testing tools are fickle and often complex to use, mandating robust agent error handling and auto-repair functionalities. These properties—partial observability, side-effecting actions, an unbounded action space, and the need for autonomous error recovery—are characteristic of a broader class of agentic SE tasks ( Yang et al. 2024 ) , and motivate studying autonomous pentesting as an instance of agentic software engineering rather than as security tooling alone. Related Harnesses Architecturally, cochise sits between minimalist agent scaffolds such as mini-SWE-agent ( Yang et al. 2024 ) and full-feature penetration-testing agents such as cAI ( Mayoral-Vilches et al. 2025 ) and incalmo ( Singer et al. 2025 ) . The closest minimalist comparison, mini-SWE-agent, lacks long-horizon trajectory management. In pilot runs against our target, GOAD, it could perform initial exploitation but did not progress to multi-stage compromise. 3. Architecture Cochise assumes an assumed breach execution-host setting: the agent begins with SSH access to a Linux host that has network reachability to the target environment and contains the tools required for the experiment. In our evaluation, this host is a Kali Linux VM connected to GOAD, and the agent’s task is to enumerate and compromise Active Directory domains. The harness itself only assumes command execution over SSH and structured observation of command outputs. Cochise does not address initial access (phishing, exploit-driven remote code execution, physical access), and we do not model human-in-the-loop adversaries. The execution host is trusted; the testbed network is the only attack surface. Figure 1. System and Agent Architecture. Orange components designate the cochise prototype, blue components are third-party tools outside of cochise’s direct control, the execution host (in yellow) allows cochise to interact with the target environment and consists of a standard Linux VM. Figure 2. Screenshot of an active cochise run, showing how cochise used certipy-ad to execute an ADCS ESC1 attack to gain Administrator level access to

Read the complete text