Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions DOI: xxxx/xxxx CCS: Security and privacy Software security engineering CCS: Software and its engineering Risk management CCS: Software and its engineering Software libraries and repositories
Chengjie Wang, Jingzheng Wu, Xiang Ling, Tianyue Luo, Chen Zhao · 2026
Read Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions DOI: xxxx/xxxx CCS: Security and privacy Software security engineering CCS: Software and its engineering Risk management CCS: Software and its engineering Software libraries and repositories by Chengjie Wang, Jingzheng Wu, Xiang Ling, Tianyue Luo, Chen Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06279.txt.
Introduction
Chengjie Wang and four co-authors measure a hidden risk in code written by language models: the library versions they specify. Generated Python code routinely imports third party libraries pinned to exact versions, and those choices can carry known flaws or break installation. No systematic study of version level risk existed before this large scale measurement on a benchmark of 1,000 programming tasks across ten models.
Models specified versions often, and between about a third and over half of tasks included at least one version with a known vulnerability. Most of those flaws were public before the model's knowledge cutoff, and models converged on the same small set of risky releases. Compatibility failures, mostly at installation, were also common. External version constraints cut both problems sharply.
Catalog record
| CREATOR | Chengjie Wang, Jingzheng Wu, Xiang Ling, Tianyue Luo, Chen Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 115 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06279
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions DOI: xxxx/xxxx CCS: Security and privacy Software security engineering CCS: Software and its engineering Risk management CCS: Software and its engineering Software libraries and repositories Chengjie Wang, Jingzheng Wu, Xiang Ling, Tianyue Luo, Chen Zhao Abstract . Large language models (LLMs) are now largely involved in software development workflows, and the code they generate routinely includes third-party library (TPL) imports annotated with specific version identifiers. These version choices can carry security and compatibility risks, yet they have not been systematically studied. We present the first large-scale measurement study of version-level risk in LLM-generated Python code, evaluating 10 LLMs on PinTrace , a curated benchmark of 1,000 Stack Overflow programming tasks. LLMs tend to specify version identifiers when directly prompted at 26.83%–95.18%, while down to 6.45%–59.19% in creating a manifest file directly. Among the specified versions, 36.70%–55.70% of tasks contain at least one known CVE, and 62.75%–74.51% of them carry Critical or High severity ratings. In 72.27%–91.37% of cases, the associated CVEs were publicly disclosed before the model’s knowledge cutoff. The statistics show all models converge on the same small set of risky release versions, indicating a systemic bias rather than isolated model error. Static compatibility rates range from 19.70% to 63.20%, with installation failure as the dominant cause. The dynamic test cases confirm the pattern by 6.49%–48.62% pass rates. Further experiments confirm that these failures are attributable to version selection rather than code quality, and that externally anchored version constraints substantially reduce both vulnerability exposure and compatibility failures. Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development. We disclosed these findings to the community of the evaluated models, and several confirmed the issue. All the code and dataset have been released for open science at https://github.com/dw763j/PinTrace . 1. Introduction Large language models (LLMs) have become a central fixture of modern software development. GitHub Copilot alone surpassed 20 million users by mid-2025, with deployment across 90% of Fortune 100 companies ( 12 ) . Copilot now contributes an average of 46% of all code to GitHub ( 11 ) , and roughly 80% of new developers on GitHub adopt it within their first week ( 12 ) . AI coding assistance has moved from an optional productivity tool to an expected part of the developer workflow ( 17 ; 73 ) , and with that shift comes new, largely unaudited risks in the code being produced. The development of modern software projects depends on third-party libraries (TPLs). Studies of large open-source ecosystems document pervasive TPL adoption, with the transitive dependency graph of a typical application spanning dozens to hundreds of packages ( 48 ; 54 ) . In traditional development, dependency version management is a deliberate, human-controlled process in which developers consult changelogs, security advisories, and compatibility notes before committing a specific release to a manifest file ( 47 ; 22 ) . When an LLM generates code, however, those same version choices are made implicitly and at scale, folded into the generated output without the developer explicitly deciding what version to use. A pinned version may carry known Common Vulnerabilities and Exposures (CVE), silently exposing every project that adopts the generated code to catalogued security risks ( 38 ) . Equally, a pinned version may be incompatible with the generated code’s API usage, rendering the snippet non-executable despite appearing syntactically correct ( 18 ; 24 ) . Both failure modes are invisible to a developer who adopts LLM-generated code without independent dependency auditing. Prior work has examined two related but orthogonal risks in LLM-generated code, yet neither addresses the version dimension. One line of research measures whether the logic of generated code contains security weaknesses, finding that LLMs produce snippets with Common Weakness Enumeration (CWE)-classified vulnerabilities at rates estimated between 12% and 65% across models and task configurations ( 8 ; 5 ; 53 ; 77 ; 10 ) . A second line documents that LLMs hallucinate non-existent package names at non-trivial rates, creating a supply-chain attack surface through which malicious actors can register the hallucinated names ( 59 ; 32 ; 33 ; 30 ) . Neither line of work addresses the version dimension: whether the specific release of an existing, correctly named library that an LLM recommends is safe to use and compatible with the generated code. This gap matters because version-level risks are structural. If LLMs converge on a small set of popular but vulnerable versions, then the same CVE propagates silently and uniformly into every project that adopts LLM-generated code referencing that library ( 38 ) , regardless of what task the code was written for. We present the first large-scale measurement study of LLM version selection behavior. We focus on the Python ecosystem because LLM-assisted Python development is pervasive ( 62 ) and PyPI’s rich vulnerability and release metadata makes version-level risk measurable ( 51 ) . We evaluate 10 LLMs on PinTrace , a curated benchmark of real-world Stack Overflow programming tasks, examining three dimensions: version specification behavior, security vulnerability exposure, and static and dynamic compatibility. Our analysis of LLM-generated code on PinTrace reveals systematic security and compatibility risks consistent across all ten evaluated models. First, version-annotation behavior is governed by format affordance rather than consistent engineering intent. Under inline-comment prompting ( inline mode), LLMs specify versions for 26.83%–95.18% of library references. Under manifest-based prompting ( explicit mode), the same models specify only 6.45%–59.19% of references. Across both modes, version choices concentrate on a narrow band of popular releases that lags each model’s knowledge cutoff by 9 to 31 months. Second, the security consequences are severe. Among tasks where LLMs specify versions, 36.70%–55.70% contain at least one version carrying a known CVE, with 62.75%–74.51% of those vulnerable versions rated Critical or High severity. In 72%–91% of cases, the associated CVEs were publicly disclosed before each model’s knowledge cutoff. Yet the same risky versions appear consistently across all model families, pointing to a shared systemic bias rather than an isolated model defect. Third, the specified versions are frequently incompatible with the generated code. Static compatibility rates fall to 19.70%–63.20% under inline prompting. Dynamic verification confirms the pattern, with pass rates collapsing to 6.49%–48.62% as version incompatibilities block execution before any test can run. Together, these three findings establish that LLM version selection introduces risks along both the security and compatibility dimensions. We conducted three additional analyses to characterize the boundaries and root causes of these risks. We first examine whether the compatibility findings hold across different Python runtime environments. Compatibility rates peak at mid-range Python versions and collapse sharply at more recent releases, with representative models dropping by more than 50 percentage points across that transition. Under explicit mode the trend reverses, with compatibility improving monotonically toward newer Python versions. This reversal confirms that LLMs systematically prefer older library releases that are compatible with legacy environments but fail under modern runtime configurations. A controlled diagnosis experiment then establishes that installation failures are attributable to version selection rather than to defects in the generated code itself. A mitigation probe shows that natural-language safety instructions yield no meaningful improvement, while external version anchoring substantially reduces vulnerability exposure.