Metamon : Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries
Hyeonseok Lee, Gabin An, Shin Yoo · 2025
Read Metamon : Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries by Hyeonseok Lee, Gabin An, Shin Yoo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02794.txt.
Introduction
Hyeonseok Lee and co-authors present METAMON, a 2025 tool that finds mismatches between code comments and real program behavior. Comments help coders grasp code, but nothing runs them, so they can drift out of date. Checking them by hand is slow and costly. METAMON mixes test generation with language model reasoning to automate the check.
First, a search-based tool builds tests that capture what the program does now. Then a language model reads those test oracles and flags ones that clash with the written spec. Metamorphic testing and self-consistency checks support the model. The team tested the tool on thousands of code and comment pairs. Such tools could keep docs honest in large code bases.
Catalog record
| CREATOR | Hyeonseok Lee, Gabin An, Shin Yoo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-02794
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Metamon : Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries Thanks: This work has been supported by the National Research Foundation of Korea (NRF) funded by the Korean government MSIT (RS-2023-00208998), and the Institute of Information Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2022-II220995). Hyeonseok Lee, Gabin An, Shin Yoo Abstract Code documentation can, if written precisely, help developers better understand the code they accompany. However, unlike code, code documentation cannot be automatically verified via execution, potentially leading to inconsistencies between documentation and the actual behavior. While such inconsistencies can be harmful for the developer’s understanding of the code, checking and finding them remains a costly task due to the involvement of human engineers. This paper proposes Metamon , which uses an existing search-based test generation technique to capture the current program behavior in the form of test cases, and subsequently uses LLM-based code reasoning to identify the generated regression test oracles that are not consistent with the program specifications in the documentation. Metamon is supported in this task by metamorphic testing and self-consistency. An empirical evaluation against 9,482 pairs of code documentation and code snippets, generated using five open-source projects from Defects4J v2.0.1, shows that Metamon can classify the code-and-documentation inconsistencies with a precision of 0.72 and a recall of 0.48. I Introduction Code documentation, such as comments or docstrings, is a human-readable description of the source code and its behavior. While languages like Java and Python provide documentation formats that can be processed by machines to produce structured documents ( PyDoc and JavaDoc , respectively), primarily code documentation aims to aid the human understanding of the code that they accompany [ 1 ] . As such, it is critical that code documentation is consistent with the behavior of the code they accompany. If the documentation is out of sync with the program behavior, developers may understand the existing code incorrectly or imprecisely, potentially leading to buggy code changes. Finding such inconsistencies between code documentation and program behavior is an important but challenging problem, due to the fact that documentation is written in natural languages whereas program behavior is described in programming langauges. Checking the semantic consistency across natural and programming languages has been primarily reserved only for human developers. However, the cost of manual inspection means that such checks cannot be performed frequently enough, resulting in inconsistencies in real-world projects [ 2 ] . Existing attempts to detect these inconsistencies either depend on rule-based approaches [ 3 , 4 , 5 , 6 ] or textual similarities [ 7 , 8 , 9 ] to make the task more tractable. Recently, LLMs have shown significant capabilities to perform logical reasoning across the barrier of natural and programming languages. In addition to successfully synthesizing code from natural language specifications [ 10 ] , LLMs can generate bug reproducing tests from bug reports written in natural language [ 11 ] , or incorporate error messages from synthesized test cases to improve the code coverage iteratively [ 12 ] . These capabilities are directly relevant to the task of checking documentation inconsistencies, and LLMs have been evaluated for the task [ 13 ] . However, existing techniques based on LLMs tend to simply prompt LLMs to spot inconsistencies between code and documentation and falls short of actually checking the dynamic program behaviour. Given that LLMs can hallucinate [ 14 ] , augmenting LLM-based inconsistency checking with a more concrete exploration of program behavior seems to be a missed opportunity. This paper presents Metamon , an automated technique that finds inconsistencies between code documentation and program behavior via the use of a search-based test data generation technique and LLMs. To capture program behavior in more detail, instead of presenting the code verbatim in prompts, Metamon uses EvoSuite [ 15 ] to generate regression test cases. These test cases not only achieve high structural coverage (and thereby expose program behavior in more depth) but also capture the current program behavior in the form of assertions that record the program output. Metamon uses LLMs to judge whether these regression test oracles are correct or not with respect to the current documentation. To further enhance its performance, Metamon adopts a couple of prompt engineering strategies. First, Metamon uses metamorphic LLM queries: given a prompt P and an answer A , Metamon will form a subsequent query prompt P^{\prime} and an expected answer A^{\prime} by transforming (P,A) with Metamorphic Relations (MRs). Second, Metamon uses Chain-of-Thoughts [ 16 ] and self-consistency [ 17 ] to improve the accuracy of the responses from the LLM. We evaluate Metamon using 9,482 pairs of Java methods and their documentation, taken from five open source projects in Defects4J [ 18 ] version 2.0.1: the dataset contains both consistent and inconsistent pairs, since we generate half of the pairs using the buggy versions. Metamon can classify the consistency between the code documentation and the program behavior captured in test cases with a precision of 0.72 and a recall of 0.48. An ablation study shows that both the metamorphic queries, and the advanced prompt engineering techniques, contribute positively to the performance. The technical contributions of this paper are as follows: • We present Metamon , an LLM-based technique that can check inconsistencies between documentation and program behavior. Metamon captures program behavior by generating regression test cases using EvoSuite, and uses metamorphic LLM queries and tailored prompt engineering techniques to enhance its performance. • We perform an empirical evaluation of Metamon based on 9,482 pairs of documentation and source code extracted from five projects in Defects4J v2.0.1. An ablation study shows that metamorphic LLM queries and self-consistency-based scoring all contribute to the final performance. • We make the implementation of Metamon , along with the dataset of 9,482 pairs of code documentation and method/test code snippets, publicly available for replicability: https://figshare.com/s/dd17b119d40d4bf3176a The rest of the paper is structured as follows: Section II provides an overview of the background and related research. Section III presents our approach, Metamon . Detailed experimental settings and research questions for evaluating Metamon are covered in Section IV , while Section V details the findings, and Section VI concludes. II Background and Related Work II-A Code-and-Documentation Inconsistency Code documentation refers to the descriptions of specifications and requirements of the program, or explanations of the source code, written with the purpose of improving code readability, facilitating easier maintenance, supporting collaboration among developers, and enhancing code reusability. However, it is common for documentation and source code to become out-of-sync over time, which can happen when only code is updated without the documentation or vice versa [ 2 ] . Analyzing the consistency between documentation and source code has traditionally been a challenging task. Some approaches have used rule-based methods to detect specific types of inconsistencies, such as issues related to locking [ 3 ] , interrupt-related concurrency bugs [ 4 ] , null value-related exceptions [ 5 ] , and identifier renaming [ 6 ] . Other techniques have treated the inconsistency detection problem as a text similarity problem, employing machine learning models [ 7 , 8 , 9 ] , which often requires a dedicated training step. Recently, LLMs have shown remarkable