Hacking CTFs with Plain Agents

Rustem Turtayev, Artem Petrov, Dmitrii Volkov, Denis Volk · 2024

Read Hacking CTFs with Plain Agents by Rustem Turtayev, Artem Petrov, Dmitrii Volkov, Denis Volk (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02776.txt.

Description

Rustem Turtayev and three colleagues test how well plain language model agents can hack in capture-the-flag contests, in a 2024 paper. These contests pose graded security puzzles, and the test set they use targets a high school level.

With simple prompting, tool use, and repeat attempts, the agents solved ninety-five percent of the benchmark tasks. That score far beats earlier results of twenty-nine and seventy-two percent. Many tasks fell in just one or two rounds of the agent's plan-and-act loop. The authors argue that today's models already pass the high school level in attack skills. The skill was there all along, they say, waiting for simple methods to draw it out.

Catalog record

CREATORRustem Turtayev, Artem Petrov, Dmitrii Volkov, Denis Volk
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-02776

Browse the full library · About the Tower of Babel