TypyBench : Evaluating LLM Type Inference for Untyped Python Repositories
Honghua Dong, Jiacheng Yang, Xun Deng, Yuhe Jiang, Gennady Pekhimenko, Fan Long, Xujie Si · 2025
Read TypyBench : Evaluating LLM Type Inference for Untyped Python Repositories by Honghua Dong, Jiacheng Yang, Xun Deng, Yuhe Jiang, Gennady Pekhimenko, Fan Long, Xujie Si (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22086.txt.
Introduction
Honghua Dong and six co-authors built a benchmark for how well language models infer types in raw Python code, in a paper published in 2025. Type hints help tools catch bugs, but most Python code lacks them. Earlier tests judged one snippet at a time. Their benchmark spans fifty complete, high-quality code repositories.
Two new scores drive the test. One grades how close an inferred type is in meaning to the true type. The other checks whether inferences agree across a whole repository. Large models did well on the first score. They often failed on nested types and on consistency across a repository. The authors say the field should pursue consistency next, not just nearly correct guesses. They release the benchmark as a base for that work.
Catalog record
| CREATOR | Honghua Dong, Jiacheng Yang, Xun Deng, Yuhe Jiang, Gennady Pekhimenko, Fan Long, Xujie Si |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22086
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TypyBench : Evaluating LLM Type Inference for Untyped Python Repositories Honghua Dong, Jiacheng Yang, Xun Deng, Yuhe Jiang, Gennady Pekhimenko, Fan Long, Xujie Si Abstract Type inference for dynamic languages like Python is a persistent challenge in software engineering. While large language models (LLMs) have shown promise in code understanding, their type inference capabilities remain underexplored. We introduce TypyBench , a benchmark designed to evaluate LLMs’ type inference across entire Python repositories. TypyBench features two novel metrics: TypeSim , which captures nuanced semantic relationships between predicted and ground truth types, and TypeCheck , which assesses type consistency across codebases. Our evaluation of various LLMs on a curated dataset of 50 high-quality Python repositories reveals that, although LLMs achieve decent TypeSim scores, they struggle with complex nested types and exhibit significant type consistency errors. These findings suggest that future research should shift focus from improving type similarity to addressing repository-level consistency. TypyBench provides a foundation for this new direction, offering insights into model performance across different type complexities and usage contexts. Our code and data are available at https://github.com/typybench/typybench . † † affiliationnotice: Equal contribution, alphabetically ordered. 1 Introduction Figure 1: Overview of TypyBench . We collect well-typed Python repositories and remove their typing information as the inputs. The outputs predicted by type inference methods are then evaluated using TypeSim and TypeCheck , where TypeSim measures the functionality similarity between predicted and human-annotated types, and TypeCheck evaluates type consistency across entire codebases through static type checking. Type inference, the ability to automatically deduce the types of variables and expressions in a program, has been a long-standing challenge in programming language research ( Raychev et al. 2015 ; Hellendoorn et al. 2018 ) . In dynamically-typed languages like Python, where explicit type annotations are optional, type inference involves analyzing code to determine appropriate type annotations that could have been written by developers. This capability has become increasingly important as codebases grow in size and complexity. The significance of type information in modern software development cannot be overstated. Type annotations serve multiple crucial purposes: (1) they enhance code clarity by making developers’ intentions explicit, (2) prevent type-related errors through early detection, (3) enable rich IDE features like autocompletion, and (4) facilitate maintenance and refactoring operations. The introduction of type hints through PEP 484 1 1 1 https://peps.python.org/pep-0484/ marked a pivotal moment for Python, acknowledging the growing importance of static typing in large-scale software development. While the benefits of type annotations are clear, manually adding them to existing codebases is time-consuming and error-prone. This challenge has sparked interest in developing automatic type inference tools like Mypy ( Lehtosalo et al. ) , Pyright ( Microsoft ) and MonkeyType ( Instagram ) , as well as learning-based algorithms ( Wei et al. 2020 ; Allamanis et al. 2020 ) . Moreover, recent advances in large language models (LLMs) have shown promising results in code understanding tasks ( Brown et al. 2020 ; Achiam et al. 2023 ; Chen et al. 2021 ) , and in type inference tasks ( Wei et al. 2023 ; Peng et al. 2023 ) with better performance than previous methods ( Shivarpatna Venkatesh et al. 2024 ) . Such tools can significantly reduce developer effort while improving code quality and maintainability. Despite these advances, current evaluation benchmarks and approaches ( Mir et al. 2021 ; Allamanis et al. 2020 ; Shivarpatna Venkatesh et al. 2024 ) for type inference methods face significant limitations. Traditional evaluation metrics rely heavily on exact matching (or up to parametric type ( Allamanis et al. 2020 ) ), which fails to capture important semantic and functional relationships between types – for instance, the functional similarity between List and Sequence , where developers may use them interchangeably. Furthermore, existing benchmarks often evaluate type inference in isolation, focusing on individual functions or files rather than considering type consistency across entire codebases. This disconnect between local correctness and global coherence makes it challenging to reliably assess the real-world effectiveness of type inference methods. To address these challenges, we introduce two novel evaluation metrics, TypeSim and TypeCheck , as illustrated in Figure 1 . TypeSim measures the functional similarity between predicted and human-annotated types by incorporating both structural relationships and type hierarchies, providing a more nuanced evaluation than exact matching. Complementarily, TypeCheck measures type consistency across entire codebases via static type checking 2 2 2 We use the number of Mypy check errors to estimate the consistency of types. , verifying that inferred types integrate coherently. We then present TypyBench , a benchmark of 50 high-quality Python repositories selected based on their type coverage, complexity, and domain diversity. Unlike existing benchmarks ( Pradel et al. 2020 ; Mir et al. 2021 ) , TypyBench emphasizes repository-level evaluation, allowing the assessment of both local type accuracy and global type consistency. Our extensive evaluation of state-of-the-art LLMs on TypyBench reveals key insights. While modern LLMs achieve decent TypeSim scores (up to 0.80), most of them struggle with TypeCheck , highlighting a gap between local type accuracy and global consistency compared to human annotations. The main contributions of this paper are: • Two novel metrics for evaluating type inference: TypeSim for measuring semantic similarity between types, and TypeCheck for assessing repository-wide type consistency. • TypyBench : A comprehensive benchmark of 50 high-quality Python repositories, designed to evaluate both local and global aspects of type inference. • An extensive empirical study of state-of-the-art LLMs on type inference, revealing critical gaps between type prediction accuracy and consistency. These contributions establish a foundation for future research in automated type inference, providing both the tools and insights needed to develop more effective type inference systems. 2 Related Work 2.1 Type Inference Methods Conventional Methods. Traditional type inference relies on static analysis and runtime tracing, as implemented in tools like Mypy ( Lehtosalo et al. ) , Pyright ( Microsoft ) , and MonkeyType ( Instagram ) . These approaches offer high precision but are limited in coverage and require explicit type annotations or runtime information. Learning-based Methods. Early learning approaches like JSNice ( Raychev et al. 2015 ) pioneered using probabilistic models to learn from existing codebases. This direction evolved to leverage various program representations, from natural language information ( Malik et al. 2019 ) to graph structures ( Hellendoorn et al. 2018 ; Wei et al. 2020 ; Allamanis et al. 2020 ; Cassano et al. 2023 ) , though struggling to handle complex type structures and rare types. LLM-based Methods. Recent work has shown that large language models can match or exceed traditional approaches in type inference tasks ( Jesse et al. 2021 ; Wei et al. 2023 ; Peng et al. 2023 ) . These methods benefit from pre-training on large code corpora and can leverage natural language understanding for improved type prediction. 2.2 Type Inference Benchmarks Previous type inference benchmarks ( Pradel et al. 2020 ; Mir et al. 2021 ; Allamanis et al. 2020 ) primarily relied on exact match accuracy for evaluation, with Typilus ( Allamanis et al. 2020 ) introducing a relaxed “Match up to Parametric Type” metric that compares only the outermost type constructors. However, these metrics still fall short of capturing full semantic similarity between type annotations.