A Study of LLMs’ Preferences for Libraries and Programming Languages
Lukas Twist, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck, Jie M. Zhang · 2025
Read A Study of LLMs’ Preferences for Libraries and Programming Languages by Lukas Twist, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck, Jie M. Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17181.txt.
Introduction
Asked to write code, large language models reach for favorite libraries and languages even when the task calls for something else. A 2025 arXiv study by Lukas Twist and co-authors measured those habits across eight models.
One popular math library appeared in up to 45 percent of answers where the reference solution did not need it. Python was the default language almost everywhere, and in speed critical setup tasks it still won 58 percent of the time, while Rust was never chosen once. The authors say the models prize familiarity over fit, and they call for training and tests that reward the right choice.
Catalog record
| CREATOR | Lukas Twist, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck, Jie M. Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-17181
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Study of LLMs’ Preferences for Libraries and Programming Languages Lukas Twist, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck, Jie M. Zhang Abstract Despite the rapid progress of large language models (LLMs) in code generation, existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make critical design choices such as which library or programming language to use. To fill this gap, we perform the first systematic study of LLMs’ preferences for libraries and programming languages when generating code, covering eight diverse LLMs. We observe a strong tendency to overuse widely adopted libraries such as NumPy ; in up to 45% of cases, this usage is not required and deviates from the ground-truth solutions. The LLMs we study also show a significant preference toward Python as their default language. For high-performance project initialisation tasks where using Python may bring more security and efficiency risks, it remains the dominant choice in 58% of cases, and Rust is not used once. These results highlight how LLMs prioritise familiarity and popularity over suitability and task-specific optimality; underscoring the need for targeted fine-tuning, data diversification, and evaluation benchmarks that explicitly measure language and library selection fidelity. Accepted to Findings of the Association for Computational Linguistics: ACL 2026. 1 Introduction Large Language Models (LLMs) have recently made rapid progress Naveed et al. (2024) , particularly excelling in code generation Jiang et al. (2024) , with extensive work on evaluating and improving LLM code accuracy, security, and efficiency Chen et al. (2024a) . However, less studied are the preferences LLMs exhibit when choosing the libraries and programming languages to use for a coding task. These preferences can have tangible effects on software performance, efficiency, maintainability, and ecosystem diversity — potentially reinforcing dominant technologies and overlooking domain-suitable alternatives. Understanding these tendencies is therefore crucial for both building trustworthy and context-aware code generation systems, and for designing benchmarks that capture not only correctness but also the rationale behind design choices in LLM-generated code. To address this gap, we present the first systematic study of library and programming language preferences across eight production-grade LLMs. We begin with libraries – vital components of modern software development Somerville (2016) – because developers often omit them from prompts Hao et al. (2024) and may not know which dependencies a task requires Larios-Vargas et al. (2020) , giving LLMs the opportunity to influence library adoption. We then consider the programming language preferences of LLMs. Many end users lack the expertise to judge whether an LLM’s choice of language is appropriate Khurana et al. (2024) ; Nguyen et al. (2024) and leave the decision entirely to the LLM, especially in end-to-end automatic software generation scenarios Sarkar and Drosos (2025) . For both libraries and programming languages, we examine LLM preferences in two key scenarios: writing code for practical tasks from widely studied benchmarks; and generating the initial code for new projects, a crucial phase where LLMs have been found to be particularly helpful Rasnayaka et al. (2024) and foundational technology choices are made. In addition, we explore whether the technologies that LLMs recommend in natural language (NL) responses match those they actually use when generating code. Our experiments lead to the following primary observations: 1) All LLMs we study heavily favour well established libraries over high-momentum alternatives . In particular, LLMs favour over-using widely adopted data science libraries even when not required for the task; for example, NumPy usage diverges with ground-truth solutions for up to 45% of tasks. 2) For programming languages, we observe a significant preference towards using Python . To our surprise, even for tasks where high performance and memory safety are critical – and therefore Python is considered suboptimal – it remains the most used language in 58% of cases. 3) LLMs do not follow their own recommendations : the technologies suggested in NL responses are a weak signal of those actually used in generated code. These tendencies likely arise from multiple factors, such as the prevalence of certain libraries in open source repositories, the distribution of programming languages included in training data Wang et al. (2024) , or post-training alignment and fine-tuning Ziegler et al. (2020) . Such preferences have mixed consequences. On the positive side, defaulting to well-known libraries and languages can speed prototyping and increase interoperability for many users. On the negative side, persistent favouritism can hurt the open-source community by marginalizing emerging projects, ultimately triggering a vicious cycle with less and less diversity as LLMs evolve with self-generated data Zhu et al. (2025) . Most critically, these biases can present a significant security threat , potentially steering users toward suboptimal and insecure implementation choices. Understanding the extent of these preferences is important, not only for assessing the potential risks and broader impacts of LLMs on the software ecosystem, but also for guiding LLM creators in improving model design, training data composition, and alignment strategies to achieve more balanced and context-aware code generation. Our contributions are as follows. 1. We present the first systematic study of LLM preferences for libraries and programming languages when generating code. 2. We highlight the importance of analysing and quantifying diversity and preference patterns in LLM-generated code, and discuss their potential implications for software development practices and the open-source ecosystem. 3. We release our code, datasets and complete results publicly via our GitHub repository, to encourage further investigation and improvement in this area: https://github.com/itsluketwist/llm-code-bias 2 Related Work Bias and unfairness in LLMs. Bias in the NL outputs of LLMs is well documented: LLMs often inherit and amplify social and representational stereotypes from their training data Bender et al. (2021) ; Nadeem et al. (2021) ; Gallegos et al. (2024) , and can preferentially surface newer content in answers Fang et al. (2025) . Social biases also appear in LLM generated code, and several datasets and frameworks now quantify code-based unfairness across demographics Liu et al. (2023b) ; Huang et al. (2024) ; Ling et al. (2025) . In contrast, technological biases – preferences for the specific technologies used in the code – are less explored. Recent work exposing provider bias (e.g., favouring specific cloud services) motivates our focus on whether analogous biases exist at the library and programming language level Zhang et al. (2025) . LLM code generation. Code generation in LLMs has been extensively studied Jiang et al. (2024) , including their ability to implement code using external or unseen libraries Liu et al. (2023a) ; Zan et al. (2023) ; Patel et al. (2024) . However, evaluation methodology remains an open problem Paul et al. (2024) . Existing approaches typically use datasets of coding tasks, but they are often limited in scope and favour popular programming languages such as Python and Java Yadav et al. (2024) . These benchmarks are also often compromised via data leakage – where the benchmarks’ tasks are included in the LLM training data – giving unfair representations of the LLMs’ abilities to complete the tasks Matton et al. (2024) ; Zhou et al. (2025) . LLM code-generation also raises several ethical issues around code reuse and provenance German et al. (2009) , as they are often trained on large public code corpora Chen et al. (2021) . Together, these issues