Hacking CTFs with Plain Agents
Rustem Turtayev, Artem Petrov, Dmitrii Volkov, Denis Volk · 2024
Read Hacking CTFs with Plain Agents by Rustem Turtayev, Artem Petrov, Dmitrii Volkov, Denis Volk (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02776.txt.
Description
Rustem Turtayev and three colleagues test how well plain language model agents can hack in capture-the-flag contests, in a 2024 paper. These contests pose graded security puzzles, and the test set they use targets a high school level.
With simple prompting, tool use, and repeat attempts, the agents solved ninety-five percent of the benchmark tasks. That score far beats earlier results of twenty-nine and seventy-two percent. Many tasks fell in just one or two rounds of the agent's plan-and-act loop. The authors argue that today's models already pass the high school level in attack skills. The skill was there all along, they say, waiting for simple methods to draw it out.
Catalog record
| CREATOR | Rustem Turtayev, Artem Petrov, Dmitrii Volkov, Denis Volk |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-02776