No Soundness in the Real World: On the Challenges of the Verification of Deployed Neural Networks

Attila Szász, Balázs Bánhelyi, Márk Jelasity · 2025

Read No Soundness in the Real World: On the Challenges of the Verification of Deployed Neural Networks by Attila Szász, Balázs Bánhelyi, Márk Jelasity (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01054.txt.

Introduction

Attila Szász, Balázs Bánhelyi, and Márk Jelasity deliver an uncomfortable verdict: today's tools that claim to prove neural networks safe do not prove what matters, in this 2025 paper. Verification aims to guarantee that a deployed network behaves within bounds. Verifiers achieve theoretical soundness, bounding the mathematical function while they compute in floating point. The catch is that the deployed network is the floating-point object, running on hardware with its rounding order and quirks.

The authors prove that the standard soundness techniques cannot bridge that gap, and that true practical soundness is much harder to compute. They then attack the verifiers directly, building adversarial networks that sense their deployment environment and misbehave where the proof is blind. Every tested verifier was fooled. The paper is a challenge to the field: safety claims must cover the machine as it runs, not the equation as written.

Catalog record

CREATORAttila Szász, Balázs Bánhelyi, Márk Jelasity
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01054

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

No Soundness in the Real World: On the Challenges of the Verification of Deployed Neural Networks Attila Szász, Balázs Bánhelyi, Márk Jelasity Abstract The ultimate goal of verification is to guarantee the safety of deployed neural networks. Here, we claim that all the state-of-the-art verifiers we are aware of fail to reach this goal. Our key insight is that theoretical soundness (bounding the full-precision output while computing with floating point) does not imply practical soundness (bounding the floating point output in a potentially stochastic environment). We prove this observation for the approaches that are currently used to achieve provable theoretical soundness, such as interval analysis and its variants. We also argue that achieving practical soundness is significantly harder computationally. We support our claims empirically as well by evaluating several well-known verification methods. To mislead the verifiers, we create adversarial networks that detect and exploit features of the deployment environment, such as the order and precision of floating point operations. We demonstrate that all the tested verifiers are vulnerable to our new deployment-specific attacks, which proves that they are not practically sound. 1 Introduction The formal verification of an artificial neural network produces a mathematical proof that the network has (or does not have) a certain important property. One important property to verify is the adversarial robustness ( Szegedy et al. 2014 ) of classifier networks, where we wish to prove that a subset of inputs are all assigned the same class label. There is a wide variety of approaches that address this problem (see Section 2 ). However, these methods invariably focus on the verification of the theoretical model of the network, that is, they seek to characterize the behavior of the full-precision computation. Most solutions carefully address floating point issues as well, but only as an obstacle in the way of verifying the theoretical model, e.g. ( Singh et al. 2019a ; Singh et al. 2018 ) . At the same time, deployment environments affect numeric computation through hardware and software features, often resulting in stochastic behavior ( Schlögl et al. 2023 ; Villa et al. 2009 ; Shanmugavelu et al. 2024 ) . This means that the verification of the theoretical model and the verification of the deployed network are different problems. Parallel to our work, a similar observation was also made by ( Cordeiro et al. 2025 ) , where this problem is referred to as the implementation gap . We formally prove that a verifier that is theoretically sound (i.e., bounds the full-precision output correctly) is not necessarily practically sound, that is, it might not bound the actual output correctly in a given deployment environment. The problem has practical implications. Recently, ( Shanmugavelu et al. 2025 ) demonstrated that adversarial inputs can be generated simply by permuting the order of associative operations in the deployment environment. We demonstrate a more severe, practically exploitable vulnerability as well: the deployed network is shown to be fundamentally different from the theoretical network because the behavior of the network can be changed arbitrarily with deployment-sensitive backdoors . Thus, the deployment environment should be a fundamental part of any verification effort because otherwise an attacker can hide potentially harmful behaviors from the verifier if enough information about the deployment environment is available. We summarize our contributions below: • We prove that verifiers that are theoretically sound are not necessarily sound for deployed networks • We demonstrate that deployed networks may differ arbitrarily from the full-precision model, with the help of backdoors triggered by features of the environment • We complement our theoretical analysis by an empirical evaluation, showing that the state-of-the-art verifiers that are theoretically sound indeed fail to correctly bound the deployments of our backdoored networks 1 1 1 Source code: https://github.com/szasza1/no_soundness 2 Related work Verification methods can be classified in various ways ( Li et al. 2023 ; Albarghouthi 2021 ; Huang et al. 2020 ; Liu et al. 2021 ) , but from a safety perspective, the most important classification categorizes algorithms based on their completeness and soundness. A verification algorithm is sound if every property it predicts to be true is true, and complete if it predicts every true property to be true. (Different terminology has also been used in the literature ( Bunel et al. 2020 ) ). Most proposed methods assume a special class of neural networks, typically ReLU networks. Sound (but not necessarily complete) methods, such as ( Singh et al. 2018 ; Singh et al. 2019a ; Zhang et al. 2018 ; Xu et al. 2021 ) , are often based on bound propagation ( Xu et al. 2020 ) and linearization. These methods deal with floating point computations as well and bound the full-precision value correctly. Methods claimed to be both sound and complete often formalize the verification problem as a satisfiability modulo theories (SMT) problem ( Katz et al. 2017 ; Ehlers 2017 ) or a mixed-integer linear programming (MILP) model ( Dutta et al. 2018 ; Tjeng et al. 2017 ) and use SMT or MILP solvers to solve them. The main disadvantages of these methods include being NP-complete ( Katz et al. 2017 ) , and relying on sophisticated solvers that might include heuristics that break soundness ( Zombori et al. 2021 ) . To mitigate the scaling issue, some methods ( Tjeng et al. 2017 ; Singh et al. 2019b ) incorporate sound (but not complete) approaches in a preprocessing phase to make the SMT or MILP models more manageable by reducing the number of integer variables. In addition, a branch-and-bound (BaB) verification framework was also proposed for sound and complete verification ( Bunel et al. 2020 ) . The basic idea is to recursively divide the verification problem into simpler subproblems, where inexpensive sound methods can prove robustness. Verifiers based on this framework mainly differ in how they solve the subproblems and how they perform the branching process. The main advantage of the BaB framework is that the verification process can be performed partially ( Xu et al. 2021 ) or fully ( Wang et al. 2021 ; Palma et al. 2021 ; Ferrari et al. 2022 ) on a GPU, significantly increasing the efficiency of the verification. The work of ( Zombori et al. 2021 ) also addresses numeric vulnerabilities in verification, but they exploit the numerical issues of the verification algorithm itself (specifically, the MILP solver). These kinds of exploitable issues are not present in sound verifiers based on bound propagation. Instead, here, we focus on exploiting the discrepancy between full-precision models and their deployment that provably mislead every sound verifier we are aware of. ( Jia & Rinard 2021 ) also address related issues. Their work can be considered an adversarial attack on verification and not a generic and provable framework we propose. It works only if the deployment and the verification algorithms use a different numerical precision, unlike our constructions. While we focus on the problems stemming from floating point arithmetic, fixed-point arithmetic should be mentioned as an interesting alternative path where quantization can be implemented with a bounded rounding error ( Lohar et al. 2023 ) , allowing for practically sound verification. Figure 1: A malicious binary classifier with a decision boundary

Read the complete text