Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models DOI: XXXXXXX.XXXXXXX Conference: 34th ACM Symposium on the Foundations of Software Engineering; June 5–9, 2026; Montreal, Canada Companion Proceedings of the 34th ACM Symposium on the Foundations of Software Engineering (FSE ’26), June 5–9, 2026, Montreal, Canada ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Maintaining software

Bar Weiss, Antonio Abu-Nassar, Adi Sosnovich, Karen Yorav · 2026

Read Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models DOI: XXXXXXX.XXXXXXX Conference: 34th ACM Symposium on the Foundations of Software Engineering; June 5–9, 2026; Montreal, Canada Companion Proceedings of the 34th ACM Symposium on the Foundations of Software Engineering (FSE ’26), June 5–9, 2026, Montreal, Canada ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Maintaining software by Bar Weiss, Antonio Abu-Nassar, Adi Sosnovich, Karen Yorav (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26100.txt.

Introduction

Weiss, Abu-Nassar, Sosnovich and Yorav move code review past free-text summaries toward structured labels. Reviewers face frequent patches, and knowing each change type, like a rename, move or logic edit, speeds triage and filtering. Current language-model tools mostly summarise or draft comments. The authors study label schemes for code changes with a two-stage pipeline. Labels first attach to diff hunks, then refinement captures structural links and semantic traits. Tests point toward multilingual review flows that machines can act on.

Catalog record

CREATORBar Weiss, Antonio Abu-Nassar, Adi Sosnovich, Karen Yorav
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43991
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26100

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models DOI: XXXXXXX.XXXXXXX Conference: 34th ACM Symposium on the Foundations of Software Engineering; June 5–9, 2026; Montreal, Canada Companion Proceedings of the 34th ACM Symposium on the Foundations of Software Engineering (FSE ’26), June 5–9, 2026, Montreal, Canada ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Maintaining software Bar Weiss, Antonio Abu-Nassar, Adi Sosnovich, Karen Yorav Abstract . Code review is a critical practice in software engineering, yet the growing scale and frequency of code patches in modern projects, together with the widespread adoption of AI code assistants, make manual review increasingly challenging. Identifying the types of changes within a patch, such as renames, moves, or logic modifications, can substantially improve review efficiency by enabling prioritization, filtering, and automation. However, existing LLM-based approaches to code review have largely focused on summarization and comment generation, leaving structured code reviews underexplored. In this paper, we present a systematic study of using large language models (LLMs) for taxonomy-based labeling of code changes in a code patch. We introduce a two-stage pipeline that assigns labels to diff hunks and then refines them to capture structural relationships and semantic attributes, such as rename propagation and type changes. Our approach employs few-shot prompting to produce language-agnostic and customizable labels, without the engineering overhead of traditional static-analysis pipelines. We evaluate four LLMs across multiple context configurations on a manually curated benchmark of natural and synthetic patches. Our best configuration achieves up to 84 % 84\% recall and 81 % 81\% precision, with high accuracy in extracting relational and attribute metadata. These results suggest that LLM-based labeling can effectively complement static analysis by enabling flexible, multilingual, and automation-friendly code review workflows. Static Analysis LLM Language Agnostic No Yes Development Effort Very High Low Maintenance Medium Low Provable Guarantees Yes No Label Customization Medium High Accuracy High ? Table 1 . Static analysis and LLM properties comparison. 1. Introduction Code review is an essential activity in software engineering: having human peers inspect changes helps catch defects early, ensures consistent standards, and fosters shared knowledge among team members ( 15 ) . Yet, as codebases grow and code patches become more frequent, review throughput and quality increasingly become bottlenecks. We use the term code patch to denote a set of code changes. In practice, such patches are often obtained from pull requests (PRs), but our formulation is independent of the surrounding review workflow. Recent advances in Large Language Models (LLMs) have transformed code-related tasks, including code completion, bug detection, and summarization ( 3 ; 10 ; 18 ) . In the context of code review, LLMs are increasingly used to summarize patches, generate review comments, and explain changes ( 16 ; 5 ; 13 ) . While these approaches aim to reduce cognitive load and improve productivity, they primarily produce free-text outputs, which are difficult to integrate into automated workflows or analytics pipelines. One promising direction is to support both human understanding and automation by labeling diff hunks according to a well-defined taxonomy of change types (e.g., rename, move, interface change, style change). Such structured output enables downstream post-processing, supports tailored workflows, and integrates naturally with dashboards and analytics systems. Traditionally, change-type identification relies on static analysis of program structure, offering strong accuracy and formal guarantees within supported domains. However, these methods are often language-specific, costly to extend, and expensive to maintain. By contrast, an LLM-based approach promises greater flexibility: it can be largely language-agnostic (assuming the model has sufficient exposure to the language), easier to adapt to new taxonomies, and faster to deploy. This flexibility comes at the cost of reduced formal guarantees and potentially less predictable accuracy, motivating a careful empirical study of its capabilities and limitations. This paper presents a study of LLMs for structured labeling of diff hunks in patches. Specifically, we investigate whether LLMs can: (1) assign taxonomy-based labels to individual diff hunks; (2) capture structure-aware relationships across hunks (e.g., renaming a function argument and all of its usages); and (3) extract semantic attributes (e.g., old/new identifier names for renaming changes, old/new types for retyping changes). To evaluate our approach, we assembled a manually curated benchmark of PRs and assessed four LLMs across multiple retrieval-context strategies. Our best configuration achieves 84% recall and 81% precision. Overall, our method complements classical static-analysis techniques by trading formal guarantees for faster adoption, multilingual support, and easier customization, while retaining structured outputs that enable practical downstream automation. We conclude by discussing trade-offs, limitations, and opportunities for extending this framework to semantic and intent-level change classification. 1 1 1 Additional experimental details and prompt templates are available in the an anonymized appendix at https://figshare.com/s/a254f611ba26b4da18a2 . 2. Related Work Traditional approaches to identifying fine-grained code changes rely on static analysis of program structure. These techniques typically operate on Abstract Syntax Trees (ASTs) and apply rule-based algorithms to detect refactoring and structural edits. Representative tools include RefactoringMiner ( 17 ) , GumTree ( 6 ) , and ChangeDistiller ( 7 ) . RefactoringMiner, for example, introduced a threshold - free AST - based matching technique capable of identifying 15 common refactoring types directly from commit histories, without requiring projects to be built. These tools demonstrate the effectiveness of static analysis for structured change detection, but remain constrained by language-specific implementations, fixed taxonomies, and the engineering effort required to adapt them to new ecosystems. LLMs have demonstrated strong capabilities in code-related tasks such as completion, bug detection, and summarization ( 3 ; 10 ; 18 ) . Code-oriented assistants like Codex ( 4 ) and Claude Code ( 1 ) leverage large-scale training on code corpora to enable few-shot and zero-shot reasoning. In the context of code review, LLMs have been applied to PR summarization, comment generation, and reviewer assistance ( 16 ; 5 ; 13 ) . While these efforts highlight the potential of LLMs to support review workflows, they primarily produce free-text outputs rather than structured representations of code changes. To the best of our knowledge, no prior work has systematically evaluated LLMs for taxonomy-based labeling of diff hunks in code patches. Existing change-classification tools rely mainly on static analysis, achieving high precision for refactoring detection but remaining language-specific and difficult to customize. In contrast, our work investigates whether LLMs can provide structured, language-agnostic labels for code edits, including cross-hunk relationships (e.g., rename propagation, code moves) and semantic attributes (e.g., old/new identifiers), opening a new direction for automation-friendly code review. Figure 1 . Example of a diff hunk with identified labeling. 3. Labeling Code Changes In this paper, we study the problem of labeling code changes. We define a set of label types \mathcal{T} , each type describing a supported category of code change (the full taxonomy is detailed in Appendix A ). Using git diff , we decompose a patch into a set of diff hunks \mathcal{H} , where each hunk corresponds to a contiguous block of changes between the old and new versions of the code. Our goal is to assign to each diff hunk in the patch all the label types from \mathcal{T} that describe the changes in it. Formally, a labeling instance is defined as a tuple, \ell=(i,h,t,p,a) , where i\in\mathbb{N} is an integer index, h\in\mathcal{H} is the

Read the complete text