HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing
Lajos Muzsai, David Imolai, András Lukács · 2024
Read HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing by Lajos Muzsai, David Imolai, András Lukács (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01778.txt.
Description
Lajos Muzsai, David Imolai, and András Lukács build a language-model agent that hacks practice targets on its own, and the yardsticks to grade it, in a 2024 paper. Their agent, HackSynth, pairs a planner that issues commands with a summarizer that digests the output, looping toward each flag. To benchmark it fairly, they assembled two hundred capture-the-flag challenges from two popular training platforms, spanning topics and difficulty.
Across open and commercial models, the agent did best with GPT-4o, beating what that model's own system card implies. Tuning experiments map how creativity settings and token budgets move the score. The authors also discuss the agent's predictability and safety, and release both the agent and the benchmarks. The work is offered as a step toward autonomous security testing with guardrails.
Catalog record
| CREATOR | Lajos Muzsai, David Imolai, András Lukács |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01778