Detecting Call Graph Unsoundness without Ground Truth DOI: XXXXXXX.XXXXXXX Conference: Conference; 2026; NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Static analysis CCS: Software and its engineering Software testing CCS: Software and its engineering Software reliability

Fangtian Zhong, Ollie Wold, Joseph Windmann · 2026

Read Detecting Call Graph Unsoundness without Ground Truth DOI: XXXXXXX.XXXXXXX Conference: Conference; 2026; NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Static analysis CCS: Software and its engineering Software testing CCS: Software and its engineering Software reliability by Fangtian Zhong, Ollie Wold, Joseph Windmann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00885.txt.

Introduction

Zhong, Wold, and Windmann show the common assumption behind Java static-analysis comparisons is wrong at its core. Studying Soot, SootUp, WALA, and Doop with precision rankings, they find precision orders break within frameworks over lambdas, reflection, and native modeling, while configuration choices interact together with algorithms. Cross-framework comparisons expose clashing ideas of what a true call graph is, challenging usual ways of judging tools.

Catalog record

CREATORFangtian Zhong, Ollie Wold, Joseph Windmann
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66803
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.00885

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detecting Call Graph Unsoundness without Ground Truth DOI: XXXXXXX.XXXXXXX Conference: Conference; 2026; NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Static analysis CCS: Software and its engineering Software testing CCS: Software and its engineering Software reliability Fangtian Zhong, Ollie Wold, Joseph Windmann Abstract . Java static analysis frameworks are commonly compared under the assumption that analysis algorithms and configurations compose monotonically and yield semantically comparable results across tools. In this work, we show that this assumption is fundamentally flawed. We present a large-scale empirical study of semantic consistency within and across four widely used Java static analysis frameworks: Soot, SootUp, WALA, and Doop. Using precision partial orders over analysis algorithms and configurations, we systematically identify violations where increased precision introduces new call-graph edges or amplifies inconsistencies. Our results reveal three key findings. First, algorithmic precision orders frequently break within frameworks due to modern language features such as lambdas, reflection, and native modeling. Second, configuration choices strongly interact with analysis algorithms, producing synergistic failures that exceed the effects of algorithm or configuration changes alone. Third, cross-framework comparisons expose irreconcilable semantic gaps, demonstrating that different frameworks operate over incompatible notions of call-graph ground truth. These findings challenge prevailing evaluation practices in static analysis and highlight the need to reason jointly about algorithms, configurations, and framework semantics when assessing precision and soundness. 1. Introduction Java remains a cornerstone of enterprise software and cloud infrastructure. Within these ecosystems, Java static analysis frameworks constitute critical infrastructure for ensuring software security, reliability, and performance ( 15 ; 14 ) . They underpin a wide range of downstream applications, including vulnerability detection ( 44 ; 45 ) , malware analysis ( 41 ) , and program optimizations ( 39 ) . Consequently, errors in Java static analysis frameworks directly compromise the reliability of the downstream decisions that depend on them. Prior work has identified a variety of observable failures in these frameworks, such as crashes ( 38 ) , memory leaks ( 8 ; 27 ) , and concurrency bugs ( 19 ) . While these failures are important, they are typically self-revealing: analysis crashes, terminates abnormally, or fails to scale, making them comparatively easier to diagnose. In contrast, semantic violations represent a more subtle and dangerous class of failures. A semantic violation occurs when an analysis completes successfully and produces plausible outputs that nonetheless violate expected semantic properties ( 24 ) . Such failures are particularly harmful in security-critical contexts. For example, a single missing call-graph edge may cause an entire data-flow path to be missed, allowing vulnerabilities such as Log4Shell to evade detection ( 5 ) . As a result, semantic violations undermine trust in automated security auditing, refactoring, and maintenance tools. Detecting such violations is challenging for several reasons. First, they are fine-grained and easily obscured by large analysis outputs. Second, they often arise not from simple implementation errors but from subtle mismatches between theoretical models and practical implementations of complex language features. Third, for real-world programs, authoritative ground truth is generally unavailable; when different analyzers disagree, it is rarely clear which result is correct. Finally, although static analysis theory provides precision and soundness guarantees, these guarantees frequently break down in practice due to engineering trade-offs and incomplete language modeling. To address both the absence of ground truth and the silent nature of semantic violations, we propose a methodology based on metamorphic testing ( 3 ) . Rather than validating analysis outputs against an external oracle, our approach checks internal semantic consistency within and across static analysis frameworks by exploiting expected relationships among analysis variants. Our key insight is that analysis algorithms and configuration settings are partially ordered by precision and soundness. For example, Class Hierarchy Analysis (CHA) ( 7 ) must conservatively over-approximate Rapid Type Analysis (RTA) ( 28 ) , which in turn must over-approximate Variable Type Analysis (VTA) ( 4 ) . Any violation of this expected ordering, such as a more precise analysis missing true-positive call edges present in a less precise variant, or introducing edges where only pruning is expected, constitutes a semantic violation, independent of any external notion of correctness. Guided by this principle, we conduct the first systematic intra-framework and cross-framework study of semantic consistency across four widely used Java static analysis frameworks. Our study spans multiple analysis algorithms, configuration settings, and their interactions. We uncover numerous previously undocumented semantic inconsistencies, many of which are triggered by modern Java language features introduced in JDK 8 and later. These findings reveal a substantial gap between theoretical expectations and practical implementations, particularly in the handling of contemporary language constructs inspired by functional programming. We emphasize that deviations from theoretical precision or soundness orders do not necessarily imply specification-level errors. Instead, we treat partial orders as pragmatic semantic contracts that capture widely accepted expectations about the relative behavior of analysis algorithms and configurations. Violations of these contracts indicate semantic inconsistencies, regardless of whether their root causes stem from implementation defects, undocumented design decisions, or incomplete modeling of modern Java features. Our approach does not assume that such partial orders must hold universally; rather, violations serve as diagnostic signals that expose semantic mismatches between theory and implementation, which are particularly valuable to surface in the absence of ground truth and enable developers to distinguish between implementation defects, undocumented design decisions, and fundamental modeling limitations. Our contributions include: • We introduce a ground-truth-free methodology for detecting semantic inconsistencies in Java static analysis frameworks by treating expected precision and soundness relationships as testable semantic contracts. This enables systematic testing in settings where labeled results are unavailable or impractical to obtain. • We generalize partial-order reasoning by unifying analysis algorithms and configuration settings within and across a single framework. This allows us to expose semantic inconsistencies that arise from algorithm–configuration and inter-framework interactions, which are invisible to existing testing approaches. • We conduct the first large-scale, cross-framework empirical study of semantic consistency across four widely used Java static analysis frameworks, uncovering previously undocumented violations and identifying recurring failure modes triggered by modern Java language features. • how to assess their capabilities in practice • How to determine the selection of algorithms and configurations when analyzing real applications 2. Background and Motivation This section motivates our work through a concrete example that highlights the challenges of testing modern Java static analysis frameworks. We then introduce the concept of partial orders over analysis algorithms and configuration settings, which we later extend to address these challenges. 2.1. Motivating Example: The Lambda Resolution Failure Figure 1. Example illustrating a lambda resolution failure in Soot. Figure 2. Example illustrating a lambda resolution failure in Soot IR. We begin with a simple yet representative example involving lambda expressions, which are widely used in modern Java programs. The example follows a common functional programming pattern in which a lambda expression is passed as a callback, as shown in Fig. 1 . From a developer’s point of view, the control flow is straightforward: the program creates an instance of a functional interface, which is later invoked to execute application-specific logic. A sound call graph should therefore resolve the invocation of the functional interface to the compiler-generated synthetic lambda method, and subsequently to the logic implemented inside the lambda. However, as shown in

Read the complete text