LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models
Daniele Cipollone, Sergey Titov, Maliheh Izadi, Egor Bogomolov, Arie van Deursen · 2026
Read LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models by Daniele Cipollone, Sergey Titov, Maliheh Izadi, Egor Bogomolov, Arie van Deursen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25402.txt.
Introduction
Real projects pin old library versions while application interfaces keep changing, so a coding model must remember several versions of the same call at once. Daniele Cipollone and colleagues test that memory in 2026 with a benchmark spanning multiple releases of widely used Python libraries. Models face tasks where the right answer depends on the stated version, and a new consistency score grades their grip on software evolution. Errors that import a call from the wrong era are the telltale failure.
Leading models proved largely blind to versions: accuracy fell on changing interfaces while stable ones scored the same across releases. Merely naming the target version in the prompt did not help at all. Supplying the relevant documentation, by contrast, raised accuracy significantly. The authors call this a systematic weakness of training on corpora that mix all eras together.
Catalog record
| CREATOR | Daniele Cipollone, Sergey Titov, Maliheh Izadi, Egor Bogomolov, Arie van Deursen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25402
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models Daniele Cipollone, Sergey Titov, Maliheh Izadi, Egor Bogomolov, Arie van Deursen Abstract Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledge of multiple API versions, not merely the latest or most common one. However, current LLMs are trained on temporally mixed corpora and lack explicit mechanisms for such version-specific reasoning, leading to anachronistic errors – calling APIs as they exist in a different library version . To systematically evaluate this phenomenon, we introduce LibEvoBench, a multi-task benchmark spanning multiple versions of widely used Python libraries, along with a new metric, the Software Evolution Understanding Score (SEUS), to measure models’ consistency when working with evolving APIs. Our results show that state-of-the-art models are largely version-oblivious: performance degrades for evolving APIs, while for stable APIs it remains the same across versions. Moreover, simply specifying the target version provides no benefit, while relevant documentation significantly boosts models’ accuracy. These findings highlight a systematic limitation of current training paradigms and motivate new approaches for temporally grounded knowledge in code generation. 1 Introduction Modern software development depends heavily on third-party libraries. Production codebases often pin hundreds of independently versioned packages, each evolving on its own release cadence with new entry points, modified signatures, and deprecated symbols. As a result, version upgrades in large projects tend to be deliberate and infrequent, constrained by compatibility requirements and the cost of migrating downstream consumers ( Kula et al. 2018 ) . As we show in Appendix A , the library version used during development may lag significantly behind the latest available release. This gap creates a version-specific development context : the correct API call, its expected parameters, and even its availability depend not on the library in general, but on the exact version specified in the project’s dependency manifest. While large language models (LLMs) have achieved remarkable results in programming tasks ( Lozhkov et al. 2024 ; Gemini 2025 ; Anthropic 2026 ) , their training procedures do not account for version specificity. These models are trained on corpora that aggregate code and documentation from many library versions simultaneously. The training objective compresses this heterogeneous material into a single set of weights, with no explicit mechanism to stratify knowledge temporally. Prior work has shown that language models struggle with temporally evolving factual knowledge ( Dhingra et al. 2022 ; Wei et al. 2025 ; Zhu et al. 2025 ) , and temporal alignment methods aim to condition retrieval on a target timestamp ( Zhao et al. 2024 ) . Version-aware code generation poses an even greater challenge: models must distinguish among multiple coexisting versions of the same library API, or risk producing anachronistic code that is valid for one version but incorrect for the requested version. To investigate how LLMs handle evolving library APIs, we introduce LibEvoBench , a multi-task benchmark spanning major versions of widely used Python libraries. LibEvoBench evaluates three tasks: API Calling (API-C), which tests contextual API use; API Identification (API-I), which tests whether models can identify the appropriate API from a description; and Signature Recall (SR), which tests version-specific signature knowledge. To pinpoint the effects of API evolution on LLM behavior, we track public APIs and signatures across versions of each library in LibEvoBench to define two API subsets: stable APIs, which remain unchanged across selected versions, and evolving APIs, which are added, removed, or modified. We then introduce the Software Evolution Understanding Score (SEUS) , which measures a model’s ability to perform consistently across both stable and evolving APIs, penalizing inconsistencies along the version axis, and capturing anachronistic errors. This metric enables ranking models based on their robustness under varying library versions in software engineering scenarios. We evaluate frontier proprietary models from ( OpenAI 2026 ) , ( Anthropic 2026 ) , and ( Gemini 2025 ) , as well as the Qwen-3.5 model family ( Team 2026 ) . Across most models, accuracy on evolving APIs drops for newer library versions while stable-API performance stays nearly flat. Even the strongest models show a persistent 7–10 point accuracy gap between operating with stable and evolving APIs. Within the Qwen-3.5 family, models of varying sizes follow similar downward trends on the evolving curve. These results suggest that the degradation reflects a structural limitation of the training process rather than a limitation of model capability or scale. We also find that most models are effectively version-oblivious. Providing the correct API documentation in the prompt yields a consistent 10–20 point improvement in API-calling accuracy, indicating that models can retrieve and use relevant information when it is made explicit. However, adding an explicit library version constraint produces no measurable improvement across any model family. Our analysis suggests that the errors mostly stem from confusion rather than fabrication: models most often produce valid but incorrect symbols from the same library. At the parameter level, anachronistic predictions—calls that would be correct for a different version of the same API—constitute a persistent portion of these errors. With this work, we make the following contributions. (i) We introduce LibEvoBench, a novel benchmark for evaluating how well LLMs can navigate evolving APIs of software libraries. (ii) We introduce SEUS , a unified metric to analyze the robustness of LLMs to changes in software libraries. (iii) We use these tools to evaluate frontier and open-weight LLMs, showing that current models remain largely version-oblivious even when given explicit version constraints. Together, these contributions provide a testbed for future researchers to develop temporally grounded and continual-learning methods for the evolving software engineering ecosystem. 2 LibEvoBench We present LibEvoBench, a benchmark that evaluates version-aware API knowledge through three complementary tasks, illustrated in Figure 1 , each targeting a different modality of knowledge. All three tasks share a common foundation: API lifecycle analysis that tracks the existence and signature of every public API across all tracked versions of a library. We first define each task (§ 2.1 ), then describe the data collection pipeline (§ 2.2 ), the lifecycle analysis and the hallucination taxonomy it enables (§ 2.3 ). 2.1 Evaluation Tasks Figure 1 : Overview of the three evaluation tasks and the API-C noise reduction levels. API-C uses real code contexts with progressively richer prompts (L0–L3). API-I isolates API identification from a redacted description. SR probes signature recall from memory. Full prompt templates and response extraction details are in Appendix F . API Calling (API-C). Given an incomplete code snippet, the model must generate the missing API call. This task evaluates model’s knowledge of the library through real API calling scenarios. Each sample originates from an open-source repository depending on a fixed library version. Every API call has been extracted and verified using a static resolution engine, which maps each call site back to a specific entry point inside the pinned library version. This traceability allows us to classify every model output according to a fine-grained hallucination taxonomy (§ 2.3 ). To obtain a cleaner signal from these samples, we introduce multiple noise reduction levels that progressively reduce ambiguity and shift the diagnostic focus by manipulating the context provided to the model. L0 presents raw code only, measuring