Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning
Chi Zhang, Haibo Qiu, Qiming Zhang, Yufei Xu, Xinbo Gao, Jing Zhang · 2026
Read Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning by Chi Zhang, Haibo Qiu, Qiming Zhang, Yufei Xu, Xinbo Gao, Jing Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11636.txt.
Introduction
Zhang and five colleagues present Seirênes, an adversarial self-play framework for LLM reasoning. Reinforcement learning with verifiable rewards advanced reasoning, yet models stay fragile amid distractions. Seirênes trains one model to build plausible distracting contexts exposing its blind spots, and to solve problems by finding the essential task. The competing goals drive a co-evolutionary curriculum. Across seven math benchmarks and scales from 4B to 30B, Seirênes gains 10 point 2, 9 point 1 and 7 point 2 points. Its 4B distractions cut top closed models by 4 to 5 points.
Catalog record
| CREATOR | Chi Zhang, Haibo Qiu, Qiming Zhang, Yufei Xu, Xinbo Gao, Jing Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74050 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11636
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning Chi Zhang, Haibo Qiu, Qiming Zhang, Yufei Xu, Xinbo Gao, Jing Zhang Abstract We present Seirênes , a self-play RL framework that transforms contextual interference from a failure mode of LLM reasoning into an internal training signal for co-evolving more resilient reasoners. While RL with verifiable rewards has significantly advanced reasoning capabilities, models can still exhibit fragility when encountering non-idealized contexts: scenarios characterized by superfluous information, tangential instructions, or incidental correlations that differ from the clean distributions typical of standard benchmarks. Seirênes harnesses this vulnerability through a parameter-shared and adversarial self-play loop. Within this framework, a single model is trained to both construct plausible yet distracting contexts that expose its own reasoning blind spots , and solve problems by discerning the essential task from these perturbations to recover the core underlying logic . By pitting these competing objectives against each other, Seirênes compels the model to move beyond superficial pattern matching and anchors its capabilities in robust underlying reasoning. This continuous interaction sustains an informative co-evolutionary curriculum as the model improves. Across seven mathematical reasoning benchmarks and model scales from 4B to 30B, Seirênes achieves average gains of +10.2, +9.1, and +7.2 points. Besides, distracting contexts produced by the 4B Seirênes model reduce the accuracy of top-tier closed-source models (GPT and Gemini) by roughly 4–5 points, revealing Seirênes’ general ability to uncover reasoning models’ blind spots. 1]School of Computer Science, Wuhan University 2]Meituan Inc 3]Independent Researchers 4]Xidian University \contribution [†]Project Leader \contribution [*]Corresponding author \code https://github.com/MiliLab/Seirenes \correspondence 1 Introduction “You will come to the Seirênes first of all; they bewitch any mortal who approaches them.” — Homer, Odyssey XII, trans. Shewring [ 1 ] Reinforcement learning with verifiable rewards (RLVR) has become the central driver of progress in LLM reasoning [ 2 ] . Propelled by a continuous streak of breakthroughs across complex reasoning domains, models have achieved competitive results in Olympiad-level mathematics and programming contests that once required years of human expertise [ 3 , 4 , 5 , 6 , 7 , 8 ] . This rapid progress has renewed interest in self-play training for reasoning models. By coupling complementary roles within a training loop, such systems can dynamically generate curricula and supervision signals, echoing the historic success of AlphaGo [ 9 ] . Recent work such as R-Zero [ 10 ] , Absolute Zero [ 11 ] , and SPICE [ 12 ] demonstrates the power of this paradigm. Existing methods typically instantiate this game at the level of task generation or solution verification: one role proposes problems, programs, or supervision, while another learns to solve them. We explore a complementary form of self-play that leaves the task itself fixed but evolves the context surrounding it. This direction is motivated by prior evidence that strong reasoners remain brittle under seemingly minor contextual changes: structural reformulations of math problems reduce accuracy by 27–31% [ 13 ] , biasing context can silently redirect chain-of-thought trajectories [ 14 ] , and irrelevant retrievals can reduce accuracy from 96% to 65% [ 15 ] . Taken together, these observations suggest that contextual interference can serve as more than an evaluation-time stress test. By exposing reasoning shortcuts that clean tasks may leave hidden, distracting contexts allow the curriculum to shift from which problems a model can solve to how it reasons through a problem. If a model can learn to generate such contexts against itself, those contexts can become an online training signal for improving the standalone reasoner. We propose Seirênes , a self-play RL framework that turns contextual interference from a failure mode of LLM reasoning into an internal training signal driven by adversarial co-evolution. The name evokes the Seirênes of Greek mythology, whose alluring songs drew sailors away from their course; in our setting, the lure takes the form of plausible but misleading contextual hints. A single shared-parameter policy plays both an Adversary , which generates plausible but misleading adversarial hints , and a Reasoner , which must solve the original problem under the perturbed context. The Adversary is rewarded for derailing the Reasoner’s solution trajectory, while the Reasoner is trained to anchor its deduction and recover the core problem logic under such interference. This adaptive training curriculum encourages Seirênes to continuously expose and solve new reasoning failures, thereby avoiding fragile reasoning shortcuts and improving robustness. Crucially, Seirênes modifies only the input context: the underlying task, verifier, and test-time interface remain unchanged, so the resulting model is optimized as a stronger standalone reasoner rather than as a special-purpose robustness module. In summary, our main contributions are as follows: • We introduce Seirênes , a self-play RL framework in which a single model co-evolves as both an Adversary that generates plausible but misleading adversarial hints and a Reasoner that learns to recover correct reasoning paths under such interference. This interaction turns contextual brittleness into an adaptive training curriculum. • We develop a parameter-shared design that makes this adversarial loop trainable end to end, combining role-aware rewards, paired clean and hint-conditioned rollouts, role-specific update streams, and an efficient orchestration strategy that keeps cost tractable at scale. • Extensive experiments show that Seirênes delivers consistent gains over both standard RL and cooperative-hint baselines on diverse mathematical reasoning benchmarks, generalizing to broader contextual perturbations. Controlled analyses attribute these gains to the dynamic adversarial loop, ruling out extra compute or static training curricula. 2 Related work RLVR for Reasoning. RLVR is the central post-training paradigm for reasoning LLMs [ 2 , 16 ] . While early studies demonstrate the efficacy of verifiable training signals [ 16 , 17 ] , DeepSeekMath [ 4 ] marks a major algorithmic milestone by introducing GRPO, which is now foundational for open-weight RLVR research. OpenAI o1 [ 5 ] and DeepSeek-R1 [ 3 ] catalyze the current wave of reasoning models by demonstrating the massive potential of test-time scaling and long-horizon RL. DeepSeek-R1-Zero, in particular, advances the field by demonstrating that pure RL can directly elicit strong reasoning capabilities from base models without supervised fine-tuning [ 3 ] . A subsequent wave of research focuses on refining these pipelines, tackling core optimization challenges like training stability, value estimation, and reproducibility [ 18 , 19 , 6 , 20 , 21 , 22 , 23 ] . As baseline reasoning capabilities rapidly expand, a long-standing question gains immediate relevance: can models move beyond being optimized by static pipelines, and actively participate in generating the challenges and signals that drive their own continuous improvement? Self-Evolving Reasoning. Recent literature explores this possibility by enabling models to synthesize their own curricula via autonomous self-play [ 10 , 11 , 12 , 24 , 25 ] . For instance, R-Zero investigates ungrounded challenger-solver co-evolution starting from zero data [ 10 ] , Absolute Zero anchors self-play in executable environments with verifiable feedback [ 11 ] , and SPICE introduces corpus-grounded self-play to sustain improvements through document-level information asymmetry [ 12 ] . Subsequent extensions incorporate minimal human anchors (e.g., R-Few [ 24 ] ) or tool integration (e.g., Agent0 [ 25 ] ) to advance this paradigm. Similar concepts also appear in multimodal domains [ 26 , 27 ] . Our work distinguishes itself by