The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution

Fei Gu,Department of Computer ScienceThe City University of Hong Kong (Dongguan)Dongguan, China [email protected], Zi LiangThe Hong Kong Polytechnic UniversityHung Hom, Hong Kong [email protected] Hongzong Li,Generative AI Research and Development CenterThe Hong Kong University · 2025

Read The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution by Fei Gu,Department of Computer ScienceThe City University of Hong Kong (Dongguan)Dongguan, China [email protected], Zi LiangThe Hong Kong Polytechnic UniversityHung Hom, Hong Kong [email protected] Hongzong Li,Generative AI Research and Development CenterThe Hong Kong University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23261.txt.

Introduction

AI coding help favors popular languages, and that edge may compound over time like interest. Fei Gu and 3 co-authors measure that asymmetry at scale. Thousands of algorithmic tasks and hundreds of framework choice tasks frame the tests. Mainstream languages and frameworks scored far higher success rates. Niche tools lagged by a substantial margin.

The pattern fits the Matthew effect, where rich ecosystems get richer support. Better AI help draws more use, which yields more training data. The authors call the gap a hidden bias in software evolution. It is not the sole driver of adoption, they note. Still, the friction for niche tech is real.

Catalog record

CREATORFei Gu,Department of Computer ScienceThe City University of Hong Kong (Dongguan)Dongguan, China [email protected], Zi LiangThe Hong Kong Polytechnic UniversityHung Hom, Hong Kong [email protected] Hongzong Li,Generative AI Research and Development CenterThe Hong Kong University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23261

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution Fei Gu,Department of Computer ScienceThe City University of Hong Kong (Dongguan)Dongguan, China [email protected], Zi LiangThe Hong Kong Polytechnic UniversityHung Hom, Hong Kong [email protected] Hongzong Li,Generative AI Research and Development CenterThe Hong Kong University of Science and TechnologyHong Kong [email protected] Jiahao Ma,MILESThe University of Hong KongHetao SZ-HK Cooperation Zone [email protected] Abstract AI-assisted programming is rapidly reshaping software development, with large language models (LLMs) enabling new paradigms such as vibe coding and agentic coding. While prior works have focused on prompt design and code generation quality, the broader impact of LLM-driven development on the iterative dynamics of software engineering remains underexplored. In this paper, we conduct large-scale experiments on thousands of algorithmic programming tasks and hundreds of framework selection tasks to systematically investigate how AI-assisted programming interacts with the software ecosystem. Our analysis reveals a striking Matthew effect: the more popular a programming language or framework, the higher the success rate of LLM-generated code . The phenomenon suggests that AI systems may reinforce existing popularity hierarchies, accelerating convergence around dominant tools while hindering diversity and innovation. We provide a quantitative characterization of this effect and discuss its implications for the future evolution of programming ecosystems. K eywords AI Programming Assistants \cdot Large Language Models (LLMs) \cdot Code Generation \cdot Vibe Coding \cdot Agentic Coding \cdot Matthew Effect \cdot Ecosystem Bias; Programming Language Diversity \cdot Framework Popularity \cdot Software Engineering Innovation \cdot Benchmarking; LeetCode Evaluation \cdot Framework-Level Analysis \cdot Ecosystem Evolution \cdot Bias Amplification. 1 Introduction Large language models (LLMs) have quickly become ubiquitous in software engineering practice, with nearly all programmers ( daigle2024survey ) utilizing AI coding tools. Tools such as GitHub Copilot, Cursor, and integrated LLM-based coding assistants now support developers in algorithmic problem solving ( yan2023closer ) , debugging, and even full-stack system construction. These advances introduce new coding paradigms: vibe coding , where developers iterate by prompting rather than typing every line, and agentic coding , where autonomous agents plan and execute end-to-end development tasks. Vibe coding democratizes software development ( gadde2025democratizing ) by lowering barriers to creation, translating conceptual intent into executable implementation. Agent-based code generation highlights the transformative potential of multi-agent systems in addressing the limitations of standalone LLMs. Agentic Coding effectively handles real-world coding challenges ( wu2024autogen ) by leveraging external tools for retrieval, achieving significant improvements ( huang2023agentcoder ) in pass rates across diverse benchmarks ( zhang2024codeagent ) . Collectively, AI Coding could be the silver bullet for software engineering. Despite these breakthroughs, long-term ecosystem-level consequences of AI programming assistance remain underexplored. This research gap is critical because biases in training data and model behavior may systematically influence which languages, frameworks, and paradigms thrive or decline. Several observations underscore why this matters. First, LLM performance is uneven across languages: high-resource ecosystems such as Python achieve disproportionately strong results, while lower-resource languages receive much weaker support. For instance, the StarCoder dataset shows Python alone accounts for nearly 40% of its training corpus, while many other languages appear only marginally ( li2023starcoder ) . Similarly, CodeGen explicitly notes that model quality varies substantially depending on training data availability, with mainstream ecosystems benefiting disproportionately ( nijkamp2022codegen ) . Second, the bias extends beyond languages to frameworks and usage patterns. AI coding assistants often over-rely on established libraries, such as NumPy, which appears in up to 48% of completions even when alternatives may be more suitable, and they also display a persistent preference for Python, which is selected 58% of the time for performance-critical tasks where other languages like Rust may be objectively better ( twist2025llms ) .Taken together, these patterns raise a central question: do AI tools genuinely empower innovation by lowering entry barriers, or do they inadvertently reinforce existing dominance hierarchies? The hypothesis we explore is that AI programming assistants exhibit a Matthew effect : “the rich get richer." This effect is rooted in the operational mechanics of LLMs, which are trained on massive datasets of publicly available code. Such dynamics risk creating lock-in effects that suppress experimentation and reduce opportunities for paradigm-shifting innovations, which is consistent with prior observations on programming language adoption and diffusion ( meyerovich2013empirical ) . Programming learners may increasingly favor languages where AI support is strongest, further consolidating existing trends ( prather2023robots ) . The empirical research on language adoption demonstrates that ecosystem factors (libraries, existing code, community size), rather than purely technical merit, strongly drive which languages gain traction, implying that model-mediated productivity gains could differentially amplify preexisting popularity patterns ( meyerovich2013empirical ) . Most existing studies of AI-assisted code generation focus on short-term, micro-level evaluations that measure model performance on narrow benchmarks or single-language datasets, which do not capture the multi-faceted complexity of real-world software engineering. If this impact is overlooked, the resulting cycle, where popular languages receive more LLM support due to their prevalence in training data, risks reducing programming ecosystem diversity. Thus, AI assistance could simultaneously lower barriers to entry while stifling long-term innovation. Contributions. This paper makes three main contributions: • We construct the first large-scale benchmark combining algorithmic programming tasks (Total 3011\times 8\times 5=120,440 ) and complex full-stack development tasks to assess AI programming assistants across languages and frameworks. • We design a controlled evaluation methodology that isolates the effect of language and framework popularity, revealing structural biases beyond aggregate accuracy metrics. • We provide the first empirical evidence that the Matthew effect emerges simultaneously at both the language and the framework levels in LLM code generation, showing how this dual-layer bias shapes software ecosystem trajectories and discussing its implications for innovation and diversity. Figure 1 presents a high-level overview of the two-tier experimental pipeline, illustrating both language-level algorithmic tasks and framework tasks Figure 1: Two-tier experimental pipeline combining algorithmic tasks and framework tasks. 2 Related Work 2.1 AI Programming Assistants Research on LLM-based programming has largely focused on improving code generation quality, prompt engineering, and usability. Systems like Codex and Copilot have demonstrated high productivity gains in everyday development. More recent models such as GPT-4, Gemini, and DeepSeek exhibit strong reasoning and multi-step planning abilities, further lowering the barrier to complex programming. Recent work has examined the capabilities of AI-assisted code generation tools across diverse benchmarks. Early evaluations using HumanEval ( chen2021evaluating ) revealed that while Copilot often produced syntactically valid code, correctness rates were low and highly correlated with the prevalence of languages in the training data ( yetistiren2022assessing ) . Although HumanEval became an early standard for evaluating LLM coding proficiency ( jiang2024survey ) , its limited number of problems restricts its applicability across all research contexts. To address this limitation, subsequent studies employed LeetCode problems. Copilot’s best accuracy in Java ( nguyen2022empirical ) , subsequent analysis extended to multiple tools, reporting that Copilot excelled in Java, ChatGPT maintained strong cross-language consistency ( batista2024code ) , and Gemini performed best in JavaScript. Larger-scale evaluations showed Copilot’s accuracy decreased with problem difficulty and varied substantially across languages ( mo2025assessing ) . The hyperscale multilingual benchmark-XCODEEVAL ( khan2024xcodeeval ) , demonstrated persistent challenges for program synthesis and translation, especially in less common languages, though its automatically collected data lacks manual verification and may introduce noise. Existing benchmarks offer valuable insights into AI

Read the complete text