Benchmarking Mythos-Linked Bug Rediscovery

Isaac David, Arthur Gervais · 2026

Read Benchmarking Mythos-Linked Bug Rediscovery by Isaac David, Arthur Gervais (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17416.txt.

Introduction

David and Gervais test whether bug-finding claims in Anthropic's April 2026 Mythos materials survive a controlled rerun. Three models get the same target files, read-only tools and three tries per task, with prompts stripped of CVE numbers, patch hints and root-cause language. Across 54 attempts only six target bugs surface, with GPT-5.5 xhigh leading at five, Claude Opus 4.7 at one and Kimi K2 at none. The main failure is early commitment to plausible wrong candidates inside the file.

Catalog record

CREATORIsaac David, Arthur Gervais
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45763
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17416

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking Mythos-Linked Bug Rediscovery Isaac David, Arthur Gervais Abstract Anthropic’s April 2026 Mythos materials combine benchmark claims with concrete bug-finding stories across OpenBSD, FreeBSD, Linux, FFmpeg, and browsers. This paper reports a controlled target-file rediscovery experiment on six public or high-confidence Mythos-linked systems tasks. Each model receives the same target file or files, read-only source tools, three repeats per task, and one manual target-matching rubric; prompts omit CVE identifiers, patch hashes, advisory text, author names, disclosure dates, and answer key root cause language. The experiment contains 54 counted model-task attempts: three models, six tasks, and three repeats, giving 18 attempts per model. GPT-5.5 xhigh achieves 5/18 target rediscoveries, covering 2/6 tasks; counting one wrong-target mpegts.c finding separately gives 3/6 distinct core bugs. Claude Opus 4.7 achieves 1/18 target rediscoveries, covering 1/6 tasks. Kimi K2 records 0/18 target rediscoveries. The dominant failure mode is early commitment to plausible alternate candidates within the assigned file: models often submit source-grounded hypotheses while missing the specific invariant corrected by public Mythos patch evidence. These results do not refute Anthropic’s undisclosed workflow, but show that under this favorable target-file scaffold, systems-specific prompting yields only six target matches across 54 counted attempts. 1 Introduction Claude Mythos Preview triggered unusual public interest because Anthropic paired benchmark improvements with concrete bug-finding stories in OpenBSD, FFmpeg, FreeBSD, Linux, and browsers [ 2 , 1 , 4 ] . The public debate around those claims quickly polarized. One side treated the anecdotes as proof of broad offensive autonomy. The other treated incomplete disclosure as evidence that the entire story was marketing. Neither position is a useful scientific null hypothesis. This paper uses one experiment. It asks whether frontier models can rediscover the six public or high-confidence Mythos-linked systems bugs when the vulnerable file is already assigned. We call this a target-file rediscovery setting: benchmark construction uses the public patch record to select the source file or files that contain the target bug, but the model only sees benchmark-safe file metadata and source access. It is intentionally not a deployable bug-hunting benchmark. It is a diagnostic: if a model cannot recover the bug after file localization is removed, then the remaining difficulty lies in intra-file coverage, invariant synthesis, candidate selection, validation, or budget. The protocol is fixed across models. Each model receives the same six tasks, the same three repeats per task, the same systems-focused worker prompt, the same read-only tools, the same validator stage, and the same manual rubric. A model enters the result table only after completing this fixed setup. We evaluate all three models. GPT-5.5 xhigh finds two target vulnerabilities: FreeBSD RPCSEC_GSS in all three attempts and FFmpeg JPEG-XS in two of three attempts. It also rediscovers the other FFmpeg MPEG-TS descriptor accounting bug once. Claude Opus 4.7 rediscovers only the FreeBSD target, once. Kimi K2 submits many candidates but does not rediscover a target bug. The contribution is a clean cross-model diagnostic, not a claim that these systems cannot find the bugs under any elicitation. Anthropic describes a larger search process with ranking, many workers, validation, and repeated attempts. We isolate an easier subproblem and find that most suite cases remain unsolved even after file localization. 2 Related Work and Public Context Our paper connects five literatures. First, Anthropic’s Mythos materials provide the core public claims: the system card, Project Glasswing launch page, and Frontier Red Team post [ 2 , 4 , 1 ] . They are unusually concrete for a frontier-lab capability release, naming systems projects, reporting campaign costs, and describing parts of the workflow. Our contribution is complementary: we turn the public portion into a reproducible rediscovery benchmark. Anthropic’s Mozilla collaboration is also important context [ 3 , 28 ] , but it concerns Claude Opus 4.6 and Firefox rather than Mythos and the systems bugs studied here. Second, the closest public response to Mythos is the Aisle essay on the jagged frontier [ 33 ] . It is a useful technical sanity check and fairly credits both the strengths and unevenness of frontier models. We frame our paper as the more controlled experimental counterpart: fixed prompts, repeated attempts, saved transcripts, source-only task inputs, and explicit target matching. Third, recent work on agentic cyber evaluation shows why scaffold design must be measured rather than assumed. CyberSecEval 2 and Cybench broaden model evaluation beyond static question answering into exploit-oriented or CTF-style settings [ 7 , 39 ] . PentestGPT, HackSynth, MAPTA, and recent architecture sweeps demonstrate that tool mediation, role decomposition, validation, and access mode can substantially change outcomes [ 17 , 29 , 15 , 16 ] . Web-focused agent studies further show both promise and limits in autonomous exploitation [ 21 , 40 ] . Our narrower study holds the scaffold fixed and focuses on six Mythos-linked systems vulnerabilities. Fourth, the smart-contract literature gives a strong example of execution-grounded security evaluation. A1, accepted at Financial Cryptography and Data Security, uses domain tools and blockchain-state execution to validate profitable exploits rather than merely score plausible vulnerability narratives [ 22 ] . That validation emphasis fits our grading philosophy, although full exploit-chain release is not suitable for our low-level C, kernel, and multimedia corpus. Fifth, our benchmark inherits lessons from program-analysis and vulnerability-dataset work. Static analysis, symbolic execution, and fuzzing systems such as Coverity, EXE, KLEE, Mayhem, Driller, AFLFast, VUzzer, and OSS-Fuzz show that real bug finding depends on search strategy, harnessing, and false-positive management [ 19 , 5 , 11 , 10 , 12 , 36 , 8 , 32 , 34 ] . Vulnerability datasets and benchmarks such as NIST SATE/Juliet, manually curated fix corpora, Big-Vul, CVEfixes, Vul4J, VulDeePecker, SySeVR, DiverseVul, PrimeVul, SecurityEval, and recent LLM vulnerability studies clarify why patch provenance, chronological splits, reproducible oracles, and realistic scoring matter [ 30 , 31 , 20 , 6 , 9 , 27 , 26 , 14 , 18 , 35 , 37 , 23 ] . General code-generation and agent benchmarks, including Codex/HumanEval, AlphaCode, ReAct, and SWE-bench, similarly motivate repeated sampling, tool use, and repository-scale context as first-class experimental variables [ 13 , 25 , 38 , 24 ] . 3 Corpus Construction The executed corpus is the same six-task set used in every reported result row. 3.1 Selection Procedure We constructed the corpus from the public Mythos claim ledger in two steps. First, we enumerated every concrete systems vulnerability that Anthropic attributed to Mythos in the Mythos public materials. Second, we kept only cases that had enough public evidence to support a hidden answer key without leaking the answer to the model: a vulnerable snapshot, a fixing patch or advisory, target file metadata, and a manual rubric covering root cause, trigger, and impact. This filter excludes many of the most dramatic Anthropic claims, including closed-source browser findings, firmware findings, and hash-committed exploit chains, because they are not yet publicly inspectable. It also excludes Firefox for a different reason. Firefox 148 contains public CVEs credited to “using Claude from Anthropic,” and Anthropic attributes that work to Claude Opus 4.6 rather than Mythos. Those cases are useful related evidence, but they form a separate non-Mythos browser suite and are outside the six systems tasks and manual grading rubrics used here. The resulting suite is

Read the complete text