Cascade : Detecting Inconsistencies between Code and Documentation with Automatic Test Generation DOI: 10.1145/3808175 Journal: PACMSE Volume: 3 FSE FSE168 7 fse26mainb-p1294-p CCS: Software and its engineering Documentation CCS: Software and its engineering Empirical software validation CCS: Software and its engineering Software testing and debugging

Tobias Kiecker, Jan Arne Sparka, Martin Reuter, Albert Ziegler, Lars Grunske · 2026

Read Cascade : Detecting Inconsistencies between Code and Documentation with Automatic Test Generation DOI: 10.1145/3808175 Journal: PACMSE Volume: 3 FSE FSE168 7 fse26mainb-p1294-p CCS: Software and its engineering Documentation CCS: Software and its engineering Empirical software validation CCS: Software and its engineering Software testing and debugging by Tobias Kiecker, Jan Arne Sparka, Martin Reuter, Albert Ziegler, Lars Grunske (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19400.txt.

Introduction

Kiecker and colleagues introduce CASCADE, a tool for spotting mismatches between code and docs while keeping false alarms low. Even small mismatches can confuse API users, introduce bugs, and raise upkeep effort, yet auto reports still need human confirmation, so false alarms waste developer time and discourage adoption. CASCADE uses large language models to draft unit tests directly from natural-language docs. Those tests are then run to expose gaps between what the docs promise and what the code does.

Catalog record

CREATORTobias Kiecker, Jan Arne Sparka, Martin Reuter, Albert Ziegler, Lars Grunske
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72972
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19400

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cascade : Detecting Inconsistencies between Code and Documentation with Automatic Test Generation DOI: 10.1145/3808175 Journal: PACMSE Volume: 3 FSE FSE168 7 fse26mainb-p1294-p CCS: Software and its engineering Documentation CCS: Software and its engineering Empirical software validation CCS: Software and its engineering Software testing and debugging Tobias Kiecker, Jan Arne Sparka, Martin Reuter, Albert Ziegler, Lars Grunske Abstract . Maintaining consistency between code and documentation is a crucial yet frequently overlooked aspect of software development. Even minor mismatches can confuse API users, introduce new bugs, and increase overall maintenance effort. This creates demand for automated solutions that can assist developers in identifying code-documentation inconsistencies. However, since automatic reports still require human confirmation, false positives carry serious consequences: wasting developer time and discouraging practical adoption. We introduce Cascade ( C onsistency A nalysis for S ource C ode A nd D ocumentation through E xecution), a novel tool for detecting inconsistencies with a strong emphasis on reducing false positives. Cascade leverages Large Language Models (LLMs) to generate unit tests directly from natural-language documentation. Since these tests are derived from the documentation, any failure during execution indicates a potential mismatch between the documented and actual behavior of the code. To minimize false positives, Cascade also generates code from the documentation to cross-check the generated tests. By design, an inconsistency is reported only when two conditions are met: the existing code fails a test, while the code generated from the documentation passes the same test. We evaluated Cascade on a novel dataset of 71 inconsistent and 814 consistent code-documentation pairs drawn from open-source Java projects. Further, we applied Cascade to additional Java, C#, and Rust repositories, where we uncovered 13 previously unknown inconsistencies, of which 10 have subsequently been fixed, demonstrating both Cascade ’s precision and its applicability to real-world codebases. † † cc-license: by 1. Introduction Inconsistencies between code and its documentation are a long-standing issue in software development ( 30 ; 39 ; 32 ) . A common oversight for developers is to update code without corresponding updates of the documentation ( 31 ) . This can hinder the efficient use of libraries ( 43 ) , increase the time required to understand a program ( 46 ) , or decrease the maintainability of software ( 20 ; 9 ) . Inconsistent documentation can also lead to the introduction of bugs in other parts of the program later in the development process ( 49 ) . Although good documentation demonstrably benefits developers ( 46 ) , the act of keeping it in sync with evolving source code is repeatedly neglected ( 50 ) . Example Figure 1 shows an example of an inconsistency in terms of a semantic mismatch between documentation and code. The startsWithAny string utility method from Apache’s commons-lang checks whether a given string starts with any string from a supplied list. The documentation erroneously claims that the method is case-insensitive. So, for example, it suggests that startsWithAny("HELLO World!", {"hell"}) would return true . In reality, the method’s implementation is case-sensitive, a fact later confirmed by the developers in a commit 1 1 1 github.com/apache/commons-lang/commit/d1a325560… that changed only this single word in the Javadoc. The inconsistency likely arose from a copy-paste error involving other case-insensitive methods in the same class. The mistake is difficult to detect through manual inspection, as the actual string comparison occurs several layers deeper in the call hierarchy in the startsWith method (invoked in line 12 ). Static approaches that just have the code and documentation as ground truth and do not execute the method, e.g., with a test case, would miss the problem. To detect these kinds of inconsistencies dynamic analysis is needed. ⬇ 1 /** 2 * Check if a CharSequence starts with any of an array of specified strings. 3 [...Shortened for better display...] 4 * @return true if the CharSequence starts with any of the the prefixes, case insensitive , 5 * or both null [...] 6 */ 7 public static boolean startsWithAny ( final CharSequence string , final CharSequence ... searchStrings ){ 8 if ( isEmpty ( string ) || ArrayUtils . isEmpty ( searchStrings )){ 9 return false ; 10 } 11 for ( final CharSequence searchString : searchStrings ){ 12 if ( startsWith ( string , searchString )) { 13 return true ; 14 } } 15 return false ; 16 } Figure 1 . Method startsWithAny from the StringUtils class in apache commons-lang. The docstring claims that the string matching is case-insensitive, while in reality it is case-sensitive. a code example This is the basic idea of our approach it consists of 4 parts. Step 1: Are documentation and code consistent? Step 2: Generate new tests consistent with the documentation. Step 3: Execute new tests against the code. Step 4: if the new tests failed we found an indicator for an inconsistency between documentation and code Figure 2 . Intuition behind Cascade . Phase 1 detects potential inconsistencies under the assumption that tests and documentation are aligned, and Phase 2 confirms this alignment. This is the basic idea of our approach it consists of 4 parts. Step 1: Are documentation and code consistent? Step 2: Generate new tests consistent with the documentation. Step 3: Execute new tests against the code. Step 4: if the new tests failed we found an indicator for an inconsistency between documentation and code Early work in detecting inconsistencies relied on specific language compiler features ( 26 ) , rule-based heuristics ( 31 ) or older Natural Language Processing (NLP) techniques ( 40 ) . With the increasing adoption of machine learning across all NLP tasks ( 24 ) , recent work has also applied it, for example in the deep-learning classifier by Panthaplackel et al. ( 28 ) . LLMs have recently shown an unprecedented ability to work with natural language, and this ability has been harnessed for many different software engineering applications ( 41 ; 23 ; 14 ) . While also promising for the inconsistency detection task, these models are notoriously unreliable. Minor changes in prompt wording can lead to divergent ( 47 ) or outright hallucinated outputs ( 13 ) . They sometimes even lack internal consistency ( 47 ) : asking an LLM a semantically similar question in a different way can produce largely different results. Example During our experiments, we asked an LLM ( gpt-4.1-mini 2 2 2 platform.openai.com/docs/models/gpt-4.1-mini ) whether the example in Figure 1 contained an inconsistency. Most of the time, it returned a clear no , completely missing the point in its explanation and focusing only on the general structure and the usage examples in the documentation. When asked the semantically similar, but oppositely framed, question of whether the method and documentation were consistent, it sometimes returned a no and in one cases gave the correct explanation by (correctly) guessing that the underlying startsWith method was case-sensitive. This highlights the volatility that pure LLM approaches often show. To conclude, traditional techniques ( 40 ; 2 ; 39 ) are narrowly scoped and LLM-based methods ( 33 ; 8 ; 16 ) are highly volatile. Both shortcomings increase the risk of false alarms, a common problem in static analysis. Static analysis tools, which are designed to

Read the complete text