LLM-Based Program Generation for Triggering Numerical Inconsistencies Across Compilers

Yutong Wang, Cindy Rubio-González · 2025

Read LLM-Based Program Generation for Triggering Numerical Inconsistencies Across Compilers by Yutong Wang, Cindy Rubio-González (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00256.txt.

Introduction

Yutong Wang and Cindy Rubio-González present LLM4FP in 2025, a framework that uses language models to generate programs that expose floating-point inconsistencies across compilers. Floating-point arithmetic rounds real numbers, and distinct compilers can round differently, which undermines numerical software. Finding programs that trigger such gaps is key to testing.

Grammar-based generation and feedback-based mutation jointly produce diverse, valid programs. Across compilers and optimization levels, LLM4FP detects nearly two and a half times more inconsistencies than the prior best tool, Varity. Most gaps involve ordinary real values, not extreme cases like NaN or infinities. The widest mismatch appears between host and device compilers. Language-model guidance widens compiler testing coverage.

Catalog record

CREATORYutong Wang, Cindy Rubio-González
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-00256

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Based Program Generation for Triggering Numerical Inconsistencies Across Compilers Yutong Wang, Cindy Rubio-González Abstract . Floating-point inconsistencies across compilers can undermine the reliability of numerical software. We present LLM4FP , the first framework that uses Large Language Models (LLMs) to generate floating-point programs specifically designed to trigger such inconsistencies. LLM4FP combines Grammar-Based Generation and Feedback-Based Mutation to produce diverse and valid programs. We evaluate LLM4FP across multiple compilers and optimization levels, measuring inconsistency rate, time cost, and program diversity. LLM4FP detects over twice as many inconsistencies compared to the state-of-the-art tool, Varity . Notably, most of the inconsistencies involve real-valued differences, rather than extreme values like NaN or infinities. LLM4FP also uncovers inconsistencies across a wider range of optimization levels, and finds the most mismatches between host and device compilers. These results show that LLM-guided program generation improves the detection of numerical inconsistencies. 1. Introduction Floating-point arithmetic is widely used in numerical software, scientific computing, and simulations. However, reasoning about its behavior remains challenging. Small rounding errors, precision issues and different compiler optimizations can all lead to unexpected behaviors ( DBL 1991 ) . For example, when the same program is compiled with different compilers and/or executed on different architectures such as CPUs and GPUs, developers may observe inconsistent results, since different compilers may generate different code for the same program ( Markstein 2008 ) . Such inconsistencies are not trivial, and detecting and understanding them is essential for building reliable numerical software. Despite their importance, finding numerical inconsistencies is challenging. First, the space of possible floating-point programs is huge, which is difficult to cover all the cases using manual or random testing. Second, such inconsistencies often happen under certain compiler optimization levels and combinations of floating-point operations which are hard to trigger. Third, numerical inconsistencies are usually small and do not cause a program crash, making it hard to notice. Therefore, there is a strong need for effective and efficient approaches to generate floating-point programs that help to discover numerical inconsistencies across compilers. To the best of our knowledge, Varity ( Laguna 2020 ) is the state-of-the-art randomized differential testing framework for generating floating-point programs that trigger numerical inconsistencies across compilers and/or optimization levels. However, Varity mainly relies on random program generation and does not make use of any domain knowledge or prior test results. Many generated programs fail to trigger any inconsistency, leading to low testing efficiency. This limits Varity ’s ability to find more subtle compiler-specific numerical inconsistencies within a reasonable budget of test programs. In a parallel line of work, Large Language Models (LLMs) have recently shown abilities in generating programs that have valid syntax and rich semantics ( Jiang et al. 2024 ) . Several tools have begun using LLMs to generate test programs for compiler fuzzing ( Xia et al. 2024 ; Yang et al. 2024 ; Munley et al. 2024 ; Gu 2023 ) . However, they mainly target general-purpose code generation, instead of focusing on floating-point programs. As a result, existing LLM-based testing approaches are not well-suited for exposing numerical inconsistencies. In this paper, we present LLM4FP , the first LLM-based framework to generate floating-point programs specifically designed to trigger numerical inconsistencies across compilers and/or optimization levels. Our insight to use LLMs is that they implicitly capture rich prior domain knowledge from a vast amount of source code “seen” during training. This benefits LLMs to generate programs with meaningful floating-point operations and diverse code patterns, which are often missed by existing random generators, and have the potential of exposing more compiler inconsistencies. To realize the potential of using LLMs for floating-point program generation, we propose two complementary program generation strategies in LLM4FP , i.e., Grammar-Based Generation and Feedback-Based Mutation. The first strategy aims to explore the broad space of floating-point programs by generating new, well-formed programs from scratch without relying on previous examples. It gives us control over the structure of the generated code, ensuring that the programs are syntactically valid and semantically meaningful. However, we observe that without feedback, this strategy may lead the LLM to generate similar program patterns which limits the diversity. In contrast, the second strategy leverages feedback by mutating previously successful programs that triggered numerical inconsistencies. By showing the LLM an effective program and instructing it to generate a different one, we encourage it to explore novel yet promising programs. Figure 1 . LLM4FP Overview We evaluate LLM4FP on a set of compilers and optimization levels. We compare against Varity , the current state-of-the-art tool, as well as two LLM-based baselines: Direct-Prompt and Grammar-Guided . To measure effectiveness, we define the inconsistency rate as the ratio of output mismatches to the total number of pairwise comparisons performed across compilers and optimization levels. The results show that LLM4FP achieves an inconsistency rate of 26.56%, more than twice that of the state-of-the-art tool Varity . This includes 4,781 total inconsistencies from a testing budget of 1,000 generated programs. Compared to baseline LLM variants, LLM4FP improves both inconsistency detection and program diversity. Notably, over 92% of the inconsistencies involve normal and subnormal numerical values instead of extreme values such as NaN or infinities, which indicates that LLM4FP is particularly effective at exposing subtle numerical inconsistencies. We further analyze inconsistency patterns across compilers and optimization levels. LLM4FP finds more inconsistencies between host and device compilers, and at lower optimization levels such as O1 and O2 , while Varity mostly detects inconsistencies only at the most aggressive level O3_fastmath . To summarize, this paper makes the following contributions: • We propose LLM4FP , the first approach that uses LLMs to generate floating-point programs specifically designed to trigger numerical inconsistencies across compilers. • We design two complementary generation strategies, i.e., Grammar-Based Generation which helps LLMs produce well-formed floating-point programs from scratch, and Feedback-based Mutation which reuses prior successful examples to explore diverse and effective program patterns. • We conduct a comprehensive evaluation across multiple compilers and optimization levels. LLM4FP outperforms existing approaches in inconsistency detection rate while overcoming the program diversity loss from other LLM-based variants, and it is especially effective at finding subtle floating-point inconsistencies. 2. Technical Approach 2.1. Overview fig. 1 shows the overall workflow of our approach, LLM4FP , for discovering floating-point inconsistencies across compilers. The first stage is to select a strategy for generating candidate programs. We propose two complementary strategies: (1) Grammar-Based Generation, which uses a given grammar specification to guide the structure of the generated programs; and (2) Feedback-Based Mutation, which randomly chooses a previously generated program that triggered numerical inconsistencies across compilers as basis to generate more programs. Once the prompt is constructed, we invoke the LLM to generate a new floating-point C program. The next stage is the Compilation Driver, which prepares the program for execution on both CPUs and GPUs. Thus this stage handles C to CUDA code translation and compilation using a configurable set of compilers and optimization flags. Finally, Differential Testing executes the programs on both host and device to detect inconsistencies in the results. If the generated program triggers any numerical inconsistencies, then it is added into the “successful” program set for potential reuse in the Feedback-Based Mutation phase. This creates a feedback loop where successful programs help guide the LLM

Read the complete text