Chain-of-Code Collapse : Reasoning Failures in LLMs via Adversarial Prompting in Code Generation
Jaechul Roh, Varun Gandhi, Shivani Anilkumar, Arin Garg · 2025
Read Chain-of-Code Collapse : Reasoning Failures in LLMs via Adversarial Prompting in Code Generation by Jaechul Roh, Varun Gandhi, Shivani Anilkumar, Arin Garg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06971.txt.
Introduction
Jaechul Roh and co-authors probe whether large language models truly reason when they write code. Their 2025 paper builds a suite of adversarial prompt changes that keep a problem's meaning but alter its surface. Changes include retelling the problem as a story, adding irrelevant constraints, reordering examples, and tweaking numbers.
Across seven hundred perturbed generations from LeetCode-style problems, some changes cut accuracy sharply, by more than forty points. Other changes oddly improved accuracy by a similar scale, though the task itself had not changed. The authors read this swing as a sign that models lean on surface patterns, not steady reasoning. They release their datasets and framework so others can test prompt robustness. The work argues for stronger checks before trusting model reasoning.
Catalog record
| CREATOR | Jaechul Roh, Varun Gandhi, Shivani Anilkumar, Arin Garg |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06971
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Chain-of-Code Collapse : Reasoning Failures in LLMs via Adversarial Prompting in Code Generation Jaechul Roh, Varun Gandhi, Shivani Anilkumar, Arin Garg Abstract Large Language Models (LLMs) have achieved remarkable success in tasks requiring complex reasoning, such as code generation, mathematical problem solving, and algorithmic synthesis — especially when aided by reasoning tokens and Chain-of-Thought prompting. Yet, a core question remains: do these models truly reason, or do they merely exploit shallow statistical patterns? In this paper, we introduce Chain-of-Code Collapse , where we systematically investigate the robustness of reasoning LLMs by introducing a suite of semantically faithful yet adversarially structured prompt perturbations. Our evaluation — spanning 700 perturbed code generations derived from LeetCode-style problems — applies transformations such as storytelling reframing, irrelevant constraint injection, example reordering, and numeric perturbation. We observe that while certain modifications severely degrade performance (with accuracy drops up to -42.1%), others surprisingly improve model accuracy by up to 35.3%, suggesting sensitivity not only to semantics but also to surface-level prompt dynamics. These findings expose the fragility and unpredictability of current reasoning systems, underscoring the need for more principles approaches to reasoning alignments and prompting robustness. We release our perturbation datasets and evaluation framework 1 1 1 https://github.com/jrohsc/Chain-of-Code-Collapse to promote further research in trustworthy and resilient LLM reasoning. Figure 1: Overview of Chain-of-Code Collapse 1 Introduction (Problem Statement) Large Language Models (LLMs) have rapidly advanced in recent years, demonstrating impressive capabilities across a range of tasks—from natural language understanding and translation to code generation and complex reasoning Brown et al., 2020 ; Chowdhery et al., 2022 ; OpenAI, 2023 ; Bai et al., 2023 ; Chen et al., 2021 . With architectural improvements and techniques like instruction tuning, chain-of-thought (CoT) prompting Wei et al., 2022 , and reinforcement learning from human feedback, modern LLMs have achieved state-of-the-art performance on benchmarks once thought to require human-level intelligence. Yet, despite their impressive breadth, LLMs remain vulnerable to adversarial inputs. Small changes in phrasing—often semantically irrelevant—can dramatically alter model behavior, especially in sensitive applications such as code generation. This vulnerability is not merely cosmetic: subtle prompt perturbations can cause models to ignore logic, omit constraints, or even produce insecure code. As prior work on DeceptPrompt Wu et al., 2023 and prompt injection attacks has shown, these models often rely on superficial patterns in prompts, rendering them brittle in the face of real-world input variability. Recent efforts have emphasized the emergent reasoning capabilities of LLMs. Through techniques like CoT prompting, models can now tackle tasks involving multi-step computation, logical inference, and algorithmic synthesis. These developments have sparked hope that LLMs may be evolving from stochastic parrots into true reasoners. However, the key question remains: do LLMs genuinely reason—or do they simulate reasoning only when cues align with familiar templates? This distinction is crucial, especially when evaluating robustness under linguistic or contextual variation. To rigorously study this question, Mirzadeh et al. (2024) proposed GSM-Symbolic ( Mirzadeh et al., 2024 ), a benchmark that perturbs math word problems using symbolic and semantic transformations while keeping their logical structure intact. Their results revealed significant accuracy drops under benign modifications—suggesting that reasoning performance is highly sensitive to surface form, even when task semantics remain unchanged. In this work, we extend this line of inquiry to the domain of code generation. We introduce Chain-of-Code Collapse (CoCC), a systematic framework for evaluating reasoning robustness in LLMs using semantically aligned yet adversarial prompt perturbations. Our perturbations span a diverse set of transformations—including storytelling, gamification, domain shifts, distracting constraints, and negation—that vary in narrative structure, lexical framing, and semantic drift. Crucially, each transformation preserves the core logic of the original task, allowing us to isolate how surface-level changes impact reasoning fidelity. As shown in Figure 1 , our methodology reveals striking differences in LLM behavior under these controlled perturbations. Some models collapse under low-preservation rewrites, while others paradoxically improve with narrative scaffolding. Unlike prior benchmarks focused on clean accuracy, our framework probes the cognitive stability of LLMs—highlighting the need for robustness evaluations that account for linguistic diversity, adversarial intent, and human-aligned prompting. In the paper, we successfully implemented six distinct perturbation methods—numeric perturbation, semantic clause injection, storytelling, gamification, domain shift, and negation. Among existing benchmarks on code generation tasks Chen et al., 2021 ; Austin et al., 2021 , we applied them to a curated set of 100 LeetCode-style problems from the LiveCodeBench dataset Jain et al., 2024 , resulting in 700 perturbed instances. We evaluated nine state-of-the-art LLMs using Pass@1 Jain et al., 2024 (adapted from LiveCodeBench), code correctness via compilation-based execution, and manual inspection. Our results showed that performance on clean prompts ranged from 95.0% (Gemini-2.5-Flash) DeepMind, 2024 to 17.0% (DeepSeek-Coder-1.3B) Guo et al., 2025 , while some perturbations like storytelling improved accuracy for specific models (e.g., +23.5% for LLaMA-3.1-Instruct Meta, 2024 ). In contrast, low-preservation attacks such as negation degraded performance by over 50% in several cases. We conducted a detailed ablation study and introduced semantic perturbation robustness as a new evaluation axis to characterize model sensitivity to benign linguistic changes. The relevance of this work is twofold. First, as LLMs become integrated into development workflows and coding assistants, ensuring their reliability under diverse and imperfect inputs becomes paramount. Second, for the broader NLP community, our findings offer a new lens through which to assess model generalization—not just in terms of accuracy, but in terms of behavioral consistency under semantic perturbation. We believe this framework can serve as a foundation for future research on trustworthy and interpretable LLM reasoning. 2 Related work The challenge of evaluating and ensuring true reasoning capabilities in Large Language Models (LLMs), particularly for complex tasks such as code generation, has been approached from several angles. While LLMs have demonstrated strong performance across multiple NLP tasks, significant challenges remain in ensuring their robustness, reliability, and the actual depth of their reasoning quality. Our work, builds upon and extends prior research in three key areas that inform our study on reasoning failures under adversarial prompt perturbations: the nature of reasoning in LLMs, specific advancements and evaluations in LLM-based code generation, and existing research on attacks against LLM reasoning. 2.1 Reasoning Large Language Models Reasoning in LLMs encompasses analytical tasks like math problem-solving, code generation, and logical inference. While traditional models often rely on pattern recognition, reasoning-enhanced models increasingly use techniques such as reasoning tokens and Chain-of-Thought (CoT) prompting Wei et al., 2022 . Reasoning tokens are designed as explicit markers within the training process to guide LLMs toward more structured reasoning, an approach that has shown improvements in handling complex reasoning challenges, especially in arithmetic and code-related tasks. CoT prompting Wei et al., 2022 and its derivatives, like Zero-shot-CoT Wang et al., 2023 and Program of Thoughts Chen et al., 2022 (which aims to separate computation from reasoning), have demonstrably enhanced multi-step reasoning. However, while CoT can improve performance on standard benchmarks, our work with CoCC investigates a critical subsequent question: whether such induced reasoning chains remain stable and effective when the problem’s surface structure is perturbed, even if the core logic is preserved. This inquiry is crucial for understanding the true depth and transferability of the elicited reasoning. Recent surveys and comprehensive reviews Mondorf and Plank, 2024 ; Prasad et al., 2023