RustEvo 2 : An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation
Linxi Liang, Jing Gong, Mingwei Liu, Chong Wang, Guangsheng Ou, Yanlin Wang, Xin Peng, Zibin Zheng · 2025
Read RustEvo 2 : An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation by Linxi Liang, Jing Gong, Mingwei Liu, Chong Wang, Guangsheng Ou, Yanlin Wang, Xin Peng, Zibin Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16922.txt.
Introduction
Linxi Liang, Jing Gong, Mingwei Liu, Chong Wang, Guangsheng Ou, Yanlin Wang, Xin Peng, and Zibin Zheng wrote this paper in two thousand twenty-five. It presents RustEvo, a fresh benchmark that tests if language models can write Rust code that fits the API of the day.
Rust shifts its interfaces fast, and models trained on old code break on new rules. The frame builds tasks from hundreds of real API changes, in four kinds: new stable forms, changed signatures, changed behavior, and dropped forms. Top models do well on stable forms but stumble on behavior shifts that leave the signature alone. Old training dates hurt, and retrieval of fresh docs claws much back.
Catalog record
| CREATOR | Linxi Liang, Jing Gong, Mingwei Liu, Chong Wang, Guangsheng Ou, Yanlin Wang, Xin Peng, Zibin Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16922
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RustEvo 2 : An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation Linxi Liang, Jing Gong, Mingwei Liu, Chong Wang, Guangsheng Ou, Yanlin Wang, Xin Peng, Zibin Zheng Abstract . Large Language Models (LLMs) have become pivotal tools for automating code generation in software development. However, these models face significant challenges in producing version-aware code for rapidly evolving languages like Rust, where frequent Application Programming Interfaces (API) changes across versions lead to compatibility issues and correctness errors. Existing benchmarks lack systematic evaluation of how models navigate API transitions, relying on labor-intensive manual curation and offering limited version-specific insights. To address this gap, we present EvoEval , a novel framework for constructing dynamic benchmarks that evaluate LLMs’ ability to adapt to evolving Rust APIs. EvoEval automates dataset creation by synthesizing 588 API changes (380 from Rust standard libraries, 208 from 15 third-party crates) into programming tasks mirroring real-world challenges. These tasks cover four API evolution categories — Stabilizations , Signature Changes , Behavioral Changes , and Deprecations — reflecting their actual distribution in the Rust ecosystem. Experiments on state-of-the-art (SOTA) LLMs reveal significant performance variations: models achieve a 65.8% average success rate on stabilized APIs but only 38.0% on behavioral changes, highlighting difficulties in detecting semantic shifts without signature alterations. Knowledge cutoff dates strongly influence performance, with models scoring 56.1% on before-cutoff APIs versus 32.5% on after-cutoff tasks. Retrieval-Augmented Generation (RAG) mitigates this gap, improving success rates by 13.5% on average for APIs released after model training. Our findings underscore the necessity of our evolution-aware benchmarks RustEvo 2 to advance LLMs’ adaptability in fast-paced software ecosystems. The framework and dataset are publicly released at ( Rus [n. d.] ) . 1. Introduction LLMs for code generation have become powerful tools for enhancing developer productivity, assisting in writing code across diverse programming languages ( Cao et al. 2025 ; Zheng et al. 2024 ; Siddiq et al. 2022 ; Zhuo et al. 2024 ; Tian et al. 2023 ; Jin et al. 2024 ) . However, these models face significant challenges in maintaining knowledge timeliness, particularly when generating code for languages with evolving APIs ( Wu et al. 2024 ; Chen et al. 2021 ; Peng et al. 2018 ; Liu et al. 2019 ; Mahmud et al. 2024b ; Sun et al. 2022 ) . Programming languages and their ecosystems continuously evolve to introduce new features, improve performance, and enhance security. Rust, known for its strong emphasis on memory safety and performance, has undergone particularly rapid evolution, with 85 release versions since its 1.0 launch in 2015. Different types of API changes, such as newly stabilized features and behavioral adjustments, are introduced with each Rust release. Despite advances in LLM-based code generation, there remains a critical gap in systematically understanding how effectively these models navigate such rapid API evolution, especially within Rust’s unique versioning model and evolution patterns ( McNamara 2021 ; Klabnik and Nichols 2019 ) . Recently, several studies have investigated LLMs’ ability to handle API evolution, particularly in established languages like Python. Wang et al. ( Wang et al. 2025 ) conduct an empirical study showing that LLMs frequently use deprecated Python APIs in code completion tasks, highlighting their difficulty in consistently adopting up-to-date APIs. Wu et al. ( Wu et al. 2024 ) evaluate LLMs’ capabilities in code generation under version-specific library requirements, revealing significant performance degradation when the required API versions differ from the predominant ones in their training data. Liu et al. ( Liu et al. 2024b ) assess LLMs’ adaptability in API updates in code generation, demonstrating that RAG with change documentation can be a promising approach to mitigating challenges posed by API evolution in LLM-based code generation. Although these existing studies have provided valuable insights, their methodologies and findings face the following limitations when applied to rapidly evolving languages like Rust. C1 LLMs Are More Likely to Face Difficulty with Rapidly Evolving APIs. Due to Rust’s rapid version evolution, its APIs frequently change for various reasons. For example, our preliminary statistics indicate that at least 495 API changes have occurred in Rust over 29 months, across 15 version updates, from July 2023 to February 2025. This high frequency of changes results in sparse or unbalanced occurrences of certain Rust API versions in open-source repositories. As a result, LLMs trained on such data are more likely to struggle with accurately using the intended Rust API versions compared to other established languages like Python. Wang et al. demonstrate that the rapid evolution of APIs leads to more frequent use of deprecated APIs by LLMs. This finding is supported by a comparative analysis of Python libraries with varying rates of evolution ( Wang et al. 2025 ) . While Python releases a new major version, such as transitioning from 3.12 to 3.13, approximately once a year ( pyt [n. d.] ) , Rust evolves much more rapidly, with a new major version, like 1.84 to 1.85, being released approximately every six weeks ( rus [n. d.] ) . Therefore, a separate evaluation of API evolution in LLM-based Rust code generation is essential for further exploration. C2 Timeliness of Evaluation Benchmarks Is Limited Due to Rapid API Evolution. LLMs typically have a knowledge cutoff date, meaning they are trained on data collected before that time. Benchmarks used to evaluate the code generation capabilities of LLMs also have their own knowledge cutoff dates. However, as Rust versions evolve and new LLMs are released, these benchmarks can become outdated and no longer accurately reflect the current capabilities of LLMs with evolving APIs. As illustrated in Figure 1 , Rust versions continue to evolve alongside LLM update cutoff dates. If an evaluation benchmark is statically constructed based on a specific snapshot of Rust APIs, the resulting evaluation likely become outdated and provide misleading insights into the latest Rust version. Figure 1 . Timeline illustrating the release dates of Rust versions and knowledge cutoff points of LLMs (2023–2025). To address these limitations, we introduce EvoEval , a novel framework for continuously constructing and evo lving benchmarks to eval uate the ability of LLM-based code generation with Rust API evolution. Note that, although our study focuses on the rapidly evolving Rust, our framework is inherently language-agnostic. The framework operates in two automated phases. In the first phase, it continuously gathers API changes from multiple sources alongside Rust version updates, including official changelogs, documentation differences, and repository implementation changes. These changes are classified into four categories: Stabilizations, Signature Changes, Behavioral Changes, and Deprecations. In the second phase, the collected API evolution changes are automatically transformed into synthetic programming tasks. Specifically, for each API change, we leverage LLMs guided by carefully designed prompts tailored to different change categories to automatically generate programming tasks. Each task includes a natural language description, a reference solution, and executable test programs. These tasks implicitly require the use of specific API versions while reflecting realistic programming scenarios. We apply our framework to recent Rust versions (1.71.0-1.84.0) and construct a dataset containing 588 API changes, with 380 (64.6%) from the official Rust language libraries and 208 (35.4%) from 15 popular third-party crates. The resulting programming tasks cover all four categories of API changes, with the distribution reflecting the