Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges

Lajos Muzsai, David Imolai, András Lukács · 2025

Read Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges by Lajos Muzsai, David Imolai, András Lukács (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02048.txt.

Introduction

Muzsai, Imolai, and Lukács train a language-model agent to crack cryptographic puzzles. Their training ground, Random-Crypto, generates fresh Capture The Flag challenges on demand, giving reinforcement learning a stream of tasks with exact right answers and many reasoning steps. They fine-tune a Python-equipped Llama model with a group-based reinforcement method inside a secure sandbox.

The trained agent solves far more unseen challenges than before, and the gains carry over to two outside tests: a public CTF benchmark spanning crypto and beyond, and a 135-question cryptography quiz. Breakdowns credit the improvement to better tool use and steadier step-by-step reasoning. The authors offer the generator as a rich practice field for security-minded agents.

Catalog record

CREATORLajos Muzsai, David Imolai, András Lukács
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02048

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges Lajos Muzsai, David Imolai, András Lukács Abstract Large Language Models (LLMs) still struggle with the structured reasoning and tool-assisted computation needed for problem solving in cybersecurity applications. In this work, we introduce random-crypto , a cryptographic Capture-the-Flag (CTF) challenge generator framework that we use to fine-tune a tool-augmented Llama-3.1-8B with Guided Reinforcement Prompt Optimisation (GRPO), allowing the agent to iteratively write and execute Python inside an isolated REPL. GRPO yields a +53% absolute jump in Pass@8 on unseen random-crypto tasks (0.35 → 0.88) and raises Majority@8 to 0.41. The fine-tuned agent also generalizes to an external dataset. On a subset of picoCTF cryptography problems, it improves Pass@8 by +13 pp. Ablations show the gains stem from more reliable tool invocation and code synthesis, rather than superficial prompt adaptation. We released the benchmark and training code on GitHub. 1 1 1 https://github.com/aielte-research/HackSynth-GRPO 1 Introduction Large language models (LLMs) have rapidly evolved from powerful text generators to sophisticated agents capable of performing complex reasoning tasks that frequently surpass human baselines [ 27 ] . This rapid advancement has spurred significant research interest in deploying LLM-based agents in critical cybersecurity roles, such as autonomous penetration testing and vulnerability detection [ 7 , 6 , 31 ] . Penetration testing has garnered considerable attention due to its practical implications for system security. This approach involves simulating cyberattacks to identify vulnerabilities and subsequently documenting exploitable weaknesses. Current LLM-based penetration testing agents primarily emphasize autonomous hacking capabilities, with their effectiveness typically assessed using Capture The Flag (CTF) challenges—structured cybersecurity puzzles designed to emulate real-world offensive and defensive scenarios [ 28 , 15 , 24 , 30 ] . However, existing benchmarks remain limited in scale and diversity [ 24 ] , posing a challenge for rigorous evaluation and comprehensive training of LLM-based cybersecurity agents. Reinforcement learning (RL) has emerged as a promising approach to enhance the capabilities of LLM agents, particularly through structured fine-tuning methods that can improve multi-step reasoning and decision-making abilities [ 2 ] . The Guided Reinforcement Prompt Optimization (GRPO) [ 25 ] method introduced by the DeepSeek team has notably demonstrated state-of-the-art performance improvements in structured reasoning tasks while maintaining computational efficiency [ 9 ] . We introduce the random-crypto benchmark—a procedurally generated dataset comprising diverse cryptographic CTF challenges. This dataset provides an effectively arbitrarily large training corpus, facilitating the systematic exploration of RL-based improvements. In this study, we use the random-crypto dataset to fine-tune a tool-augmented Llama-3.1-8B model [ 13 ] . We demonstrate that GRPO, not only enhances agent performance in cryptographic problem-solving but also improves the reliability and effectiveness of tool usage. Specifically, our paper makes the following contributions: 1. We introduce a novel, programmatically generated cryptographic benchmark ( random-crypto ) for training and evaluation of cybersecurity-focused language models. 2. We demonstrate performance enhancements in LLM agents through GRPO-based reinforcement learning, particularly highlighting improvements in tool-augmented interactions. 3. We validate the generalization of our trained agents on an external dataset (picoCTF), underscoring the robustness of our approach. 2 Background 2.1 CTF Challenges CTF challenges are widely used to train and evaluate cybersecurity skills by simulating real-world attack scenarios in sandbox environments. The goal is typically to discover a hidden text string, called a flag, embedded in vulnerable applications, binaries, or websites. CTF challenges encompass a wide array of cybersecurity domains, including: web exploitation, cryptography, reverse engineering, forensics, and binary exploitation. Cryptographic challenges in particular revolve around exploiting weaknesses or logical flaws in encryption and encoding methods. These challenges commonly involve classical ciphers (e.g., Caesar, Vigenère, substitution), symmetric-key encryption (e.g., AES, DES), and public-key encryption schemes (e.g., RSA, ECC). Participants may also encounter tasks involving hashing algorithms, digital signatures, and custom cryptosystems, which often require creative approaches and deep theoretical understanding to solve. 2.2 LLM Agents in Cybersecurity Recent advancements in large language models (LLMs) have sparked significant interest in their application within cybersecurity, particularly in penetration testing and security assessments. Researchers have adapted Capture The Flag (CTF) challenges as standardized benchmarks [ 28 , 15 , 30 , 24 ] , to systematically evaluate the cybersecurity capabilities of LLM-based agents. Initial evaluations revealed that general-purpose LLMs struggled to effectively handle specialized cybersecurity tasks due to their inherent inability to interact directly with computing environments [ 26 , 28 , 15 , 4 ] . In other domains, to bridge this interaction gap, newer architectures introduced tool augmentation, empowering LLM agents to invoke external tools or APIs. Frameworks like ReAct (Reason+Act) demonstrated substantial improvements by integrating logical reasoning with actionable tool use [ 29 ] . OpenAI’s ChatGPT further extended this approach through a plugin/function-calling API, enabling deterministic JSON tool calls [ 16 ] . Additionally, Anthropic’s Model Context Protocol (MCP) provides a structured interface for models to interact seamlessly with external data sources and tools, facilitating complex cybersecurity tasks that require real-time computational support [ 1 , 11 ] . These tool-augmented LLM agents have markedly advanced the state-of-the-art. However, they have yet to be fully utilized in autonomous penetration testing. 2.3 Reinforcement Learning to Enhance LLM Reasoning Reinforcement learning (RL) has become a central technique for improving the reasoning abilities of LLMs, particularly in tasks requiring multi-step problem solving and generalization. The standard Reinforcement Learning by Human Feedback (RLHF) paradigm fine-tunes models based on human preference signals, typically using Proximal Policy Optimization (PPO) [ 23 ] , to align model outputs with desired behaviors [ 19 ] . Recent work has shown that traditional RLHF, while effective in improving helpfulness and safety, often falls short in domains that require structured, multi-hop reasoning like math, programming, and cybersecurity. To address this, specialized algorithms have emerged, such as Group Relative Policy Optimization (GRPO) [ 25 ] . GRPO is a variant of PPO tailored for reasoning-intensive tasks: instead of relying on scalar preference labels, it compares multiple candidate outputs and assigns relative rewards within a group. This relative evaluation strategy encourages diversity and robustness in generated solutions, helping the model refine its internal decision-making process. GRPO has been successfully applied to train DeepSeek-R1 [ 9 ] , achieving state-of-the-art performance on mathematical reasoning and competitive coding benchmarks. Its low computational overhead and compatibility with low-rank adaptation methods such as QLoRA [ 5 ] make it an attractive choice for resource-constrained environments. 2.4 PicoCTF Evaluation Subset To evaluate the transferability of the trained models beyond random-crypto , we curated a subset of the picoCTF benchmark [ 15 ] . Challenges were selected based on two criteria: (i) the challenge is tagged as cryptography , and (ii) the challenge provides a self-contained input, either as plaintext or as files that can be concatenated into an ASCII string. This excludes tasks requiring interactive services (e.g., netcat ) or non-ASCII artifacts (e.g., images). Using these filters, we obtained 8 cryptography challenges for evaluation. 3 Methods 3.1 Random-Crypto Benchmark Dataset We introduce the random-crypto benchmark, a systematically created, human-verified dataset tailored for training and evaluating cybersecurity-focused language models. The benchmark consists of cryptographic Capture The Flag (CTF) challenges designed to test a solver’s ability to exploit vulnerabilities in cryptographic schemes and protocols. Each challenge requires recovering a hidden flag in the form flag{...} by leveraging

Read the complete text