How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions
Ningzhi Tang, Chaoran Chen, Gelei Xu, Yiyu Shi, Yu Huang, Collin McMillan, Tao Dong, Toby Jia-Jun Li · 2026
Read How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions by Ningzhi Tang, Chaoran Chen, Gelei Xu, Yiyu Shi, Yu Huang, Collin McMillan, Tao Dong, Toby Jia-Jun Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29442.txt.
Introduction
Ningzhi Tang studies coding-agent failures in developer sessions. Benchmark analyses miss developer experience. The study covers over twenty thousand sessions from over sixteen hundred repositories across IDE and CLI workflows. Misalignment means a breakdown visible through developer pushback, tagged by form, cause, cost, and resolution. Seven recurring forms emerge: reading projects, interpreting intent, following rules, bounding actions, coding, and reporting progress. About nine tenths of episodes cost effort and trust rather than irreversible damage, yet over nine tenths of resolutions still need explicit user correction.
Catalog record
| CREATOR | Ningzhi Tang, Chaoran Chen, Gelei Xu, Yiyu Shi, Yu Huang, Collin McMillan, Tao Dong, Toby Jia-Jun Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77460 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29442
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions Ningzhi Tang, Chaoran Chen, Gelei Xu, Yiyu Shi, Yu Huang, Collin McMillan, Tao Dong, Toby Jia-Jun Li Abstract AI coding agents increasingly act directly within software environments, yet existing analyses of their failures rely on benchmark trajectories that miss how developers actually experience misalignment. We present an observational study of 20,574 coding-agent sessions from 1,639 repositories across IDE and CLI workflows. We operationalize misalignment as a breakdown made visible through developer pushback, and annotate each episode along four axes: form, cause, cost, and resolution. We identify seven recurring forms, spanning how agents read projects, interpret developer intent, follow rules, bound their actions, implement and execute code, and report progress. 90.50% of episodes impose effort and trust costs rather than irreversible system damage, yet 91.49% of visible resolutions still require explicit user correction. Misalignment patterns also differ across IDE and CLI settings, persist across adjacent sessions, and shift over time: while overall rates decline, constraint violations and inaccurate self-reporting grow in share. Our findings inform the design of training, evaluation, and interfaces for keeping coding agents aligned with real developer workflows. 1 Introduction AI coding agents have moved beyond text generation to act directly in software environments, handling multi-turn development tasks that involve file edits, command execution, and sustained communication with developers. This shift changes what alignment requires: rather than correctness on isolated tasks, agents must stay aligned with developer intent as both the task and that intent evolve across turns. In practice, this proves difficult: developers rarely give agents a complete specification upfront; instead, they refine their requests turn by turn, often changing direction as they see what the agent produces Tang et al. 2026 . The resulting friction is measurable: users interrupt agents mid-turn in 5% of interactions and push back against outputs in 41% of turns Baumann et al. 2026 . However, systematic empirical characterizations of this friction remain limited. The closest existing work studies agent failures from within the agent itself. For example, Cemri et al. 2026 and Zhang et al. 2025 analyze execution traces on controlled benchmarks to classify where agents go wrong and which component in a pipeline is responsible. These analyses are rigorous on their own terms, but benchmark trajectories are generated under pre-specified tasks with no real developer in the loop. As a result, they cannot capture misalignment as developers experience it: not only whether an agent task succeeded or failed, but what form the divergence took, why it occurred, and how developers detected and corrected it across real sessions. Closing this gap requires interaction logs from naturalistic sessions rather than benchmark trajectories. Without them, efforts to improve coding agents lack empirical grounding for where and how alignment breaks down in practice. To bridge this gap, we present, to the best of our knowledge, the first large-scale characterization of developer-agent misalignments in the wild. We define misalignment as observable breakdowns in developer-agent collaboration that surface through developer correction or pushback in conversational logs, and scope our analysis to two proximal alignment goals: instructions (what developers explicitly ask for) and intentions (what they actually want) Shen et al. 2024 . We analyze two complementary datasets of 20,574 real IDE and CLI coding-agent sessions across 1,639 repositories Tang et al. 2026 ; Baumann et al. 2026 , and develop an LLM-based extraction pipeline with a second-stage evidence filter that removes claims unsupported by the conversation, yielding 16,118 evidence-grounded episodes with a human-evaluated precision of 0.93. We characterize each episode along four axes (symptom, cause, outcome, and resolution) using an LLM judge validated against human experts (inter-rater agreement 0.83 ; LLM judge accuracy 0.81 ), and organize the analysis around four research questions spanning misalignment forms and causes (RQ1), outcomes and resolution patterns (RQ2), variation across IDE and CLI modalities (RQ3), and structural and temporal effects (RQ4). We highlight the following findings. First , we identify seven recurring symptom categories and seven cause categories that characterize developer-agent misalignment, spanning how agents read the project, interpret developer intent, follow stated rules, bound their actions, implement and execute work, and report their progress. Second , 90.50% of episodes impose effort and trust costs rather than irreversible system damage; visible resolution occurs in only 9.33% of episodes, and 91.49% of these require explicit developer pushback. Third , misalignment differs systematically across modalities: CLI sessions are more prone to constraint violations, with damage extending to project and external state, whereas IDE sessions more often surface faulty implementations and underspecified instructions confined to task state. Finally , misalignment persists across adjacent sessions in the same repository; its overall rate declines over time, but constraint violations and inaccurate self-reporting grow in share, suggesting coding agents need improvements beyond implementation accuracy. 2 Related Work 2.1 Coding Agents in Developer Workflows AI coding agents mark a clear shift from earlier code-generation tools. Unlike inline autocompletion or single-turn chat assistants, they combine language reasoning, tool use, and sub-agent invocation to operate autonomously within live codebases Jimenez et al. 2024 ; Yang et al. 2024 ; Li et al. 2025 . Agent sessions leave interaction traces in public repositories, making real-world usage increasingly observable. Two large-scale studies provide the initial empirical foundation. Tang et al. 2026 analyzes 11,579 IDE sessions from Cursor and GitHub Copilot across 1,300 repositories, finding that developers rarely specify tasks upfront; instead, they refine requests progressively, redistribute cognitive work such as comprehension and validation to the agent, and actively manage agent behavior throughout a session. Baumann et al. 2026 extend this picture to CLI-based workflows, analyzing 6,000 sessions involving more than 355,000 tool calls and finding that only 44% of agent-written code survives into final commits. Together, these studies establish that developer-agent interaction is iterative, corrective, and marked by persistent friction. However, neither characterizes the forms that friction takes, where it originates, or how it is resolved. 2.2 Failure Analysis of Coding Agents The current dominant approach to understanding agentic failures analyzes agent-internal trajectories on predefined controlled benchmarks. Cemri et al. 2026 introduces MAST , a taxonomy of 14 failure modes derived from 1,642 execution traces across five multi-agent frameworks. Zhang et al. 2025 extends this line of work by attributing failures to specific agents and steps within multi-agent pipelines, while related studies examine behavioral patterns in agent trajectories Majgaonkar et al. 2025 ; Mehtiyev and Assunção 2026 . A separate line of work infers agent failures from downstream artifacts, such as whether agent-written code is accepted into software projects Ehsani et al. 2026 ; Alam et al. 2026 , surfacing useful signals about which kinds of agentic contributions succeed or fail. However, these studies primarily illuminate either how agents fail internally or how their outputs fare downstream, leaving the developer’s real-time corrective process unexamined. 2.3 Human-AI Alignment Aligning AI agents with human intent has primarily been approached through training-time interventions. Reinforcement learning from human feedback (RLHF) Ouyang et al. 2022 optimizes model behavior against preference signals collected from human comparisons, while reinforcement learning with verifiable rewards (RLVR) Lambert et al. 2024 ; Guo et al. 2025 sidesteps human annotation by using programmatic outcomes, e.g., whether generated code passes tests, as supervision signals. More recent work