Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinement

Hengjie Liu, Zhenya Zhang (✉) {}^{\text{({\char 0\relax})}}, Jianjun Zhao · 2026

Read Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinement by Hengjie Liu, Zhenya Zhang (✉) {}^{\text{({\char 0\relax})}}, Jianjun Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14294.txt.

Introduction

Liu, Zhang and Zhao propose a transformer checking approach with better precision via ReLU-catalyzed abstraction refinement. Transformer inferences involve complex dot products in self-attention layers, making checks extremely difficult. Current methods bound output ranges well, but sacrifice precision and cause frequent false alarms. ReLU gives a precise non-linear bound for dot products, letting the team use convex relaxation work for ReLU to derive precise bounds. Rule-based and optimization-based classics are extended to them, yielding two new frameworks for efficient and precise checks.

Catalog record

CREATORHengjie Liu, Zhenya Zhang (✉) {}^{\text{({\char 0\relax})}}, Jianjun Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79990
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinement Hengjie Liu, Zhenya Zhang (✉) {}^{\text{({\char 0\relax})}}, Jianjun Zhao Abstract Formal verification of transformers has become increasingly important due to their widespread deployment in safety-critical applications. Compared to classic neural networks, the inferences of transformers involve highly complex computations, such as dot products in self-attention layers, rendering their verification extremely difficult. Existing approaches explored over-approximation methods by constructing convex constraints to bound the output ranges of transformers, which can achieve high efficiency. However, they may sacrifice verification precision, and consequently introduce significant approximation error that leads to frequent occurrences of false alarms. In this paper, we propose a transformer verification approach that can achieve improved precision. At the core of our approach is a novel usage of ReLU, by which we represent a precise but non-linear bound for dot products such that we can further exploit the rich body of literature for convex relaxation of ReLU to derive precise bounds. We extend two classic approaches to the context of transformers, a rule-based one and an optimization-based one, resulting in two new frameworks for efficient and precise verification. We evaluate our approaches on different model architectures and robustness properties derived from two datasets about sentiment analysis, and compare with the state-of-the-art baseline approach. Compared to the baseline, our approach can achieve significant precision improvement for most of the verification tasks with acceptable compromise of efficiency, which demonstrates the effectiveness of our approach. 1 Introduction Over the past decade, the rapid development of deep learning has driven unprecedented breakthroughs in different application domains. As a ground-breaking milestone, transformers [ 27 ] , empowered by the self-attention mechanism, have brought revolutionary performance advances to multiple fields, such as computer vision and natural language processing (NLP). However, similar to other types of neural networks, transformers are notoriously vulnerable to adversarial perturbations, which brings significant doubts on their trustworthiness. For instance, synonym substitution [ 21 , 1 ] has proved to be an effective attack method in NLP that can fool the model by replacing some words of an input sentence with their synonyms. Given the increasing deployment of transformers in safety-critical applications (e.g., autonomous driving, healthcare), ensuring their robustness in noisy and sophisticated environments has become an urgent demand. Verification is a preferred approach that can formally prove the safety and robustness of systems, thereby alleviating the concerns about their misbehaviors. In the context of neural networks, it has been extensively studied (see a survey [ 19 ] and the competition report [ 6 ] ) with well-established methodological foundations. The challenge mainly involves how to deal with their non-linear components , e.g., activation functions in classic feed-forward neural networks. Due to this, computing an exact bound of neural network outputs often incurs exponential time complexity and thus is not scalable to real-world models. By contrast, abstract interpretation is an approach that constructs convex over-approximation of their outputs and assesses whether the approximated outputs satisfy the desired property—as long as the over-approximated outputs satisfy the property, the original outputs must also satisfy the property, thereby verifying the neural networks. Owning to the convexity of the approximated output bounds, these approaches are much more efficient and thus preferable in practice. Overview of Related Work Despite the prosperity of research in classic neural network verification, verification of transformers still remains at a very early stage. Nevertheless, standing on the shoulders of classic verification, the research community get straight to abstract interpretation [ 23 , 3 , 35 ] , which is more scalable to transformers of large sizes. For example, as the state-of-the-art, Shi et al. [ 23 ] extend the idea of linear relaxation from classic neural network verification, and devise different linear bounds for different non-linear components of transformers, such as the dot product operations in self-attention layers. Compared to the verification approaches that bound transformer outputs exactly (e.g., [ 18 ] ), computation of convex over-approximation often requires polynomial time complexity, so they are much faster and more scalable, and therefore they are feasible to be used to handle complex transformers in real world. Completeness Issues of Abstract Interpretation As an intrinsic issue of abstract interpretation, if the over-approximated outputs violate the given property, it does not imply that the original outputs also violate the property, and in this case, the reported violation (i.e., the violation by the over-approximated outputs) could be a false alarm . This is known as the completeness issue [ 19 ] , and essentially, it arises from the error introduced during the approximation of the transformers’ non-linear components. In particular, we show that existing over-approximation approaches are often suboptimal and thus may introduce significant approximation error and bring many false alarms. Given that verification is typically expensive, frequent occurrences of false alarms can lead to substantial waste of time and resources, highlighting the need for more precise approximation methods. Key Idea of Abstraction Refinement While there are various non-linear operations in transformers, dot products happen most frequently in matrix multiplications of self-attention layers and pose unique challenges that do not exist in classic neural network verification. In [ 23 ] , the main contribution involves the design of a provably correct planar bound for dot products, which is given in terms of a rigid strategy that always covers one side of the hyperbolic paraboloid derived from the dot product (see Fig. 2(b) for illustration). While it is a sound choice, it is not the unique one, e.g., there can be a dual bound that covers another side of the hyperbolic paraboloid (see Fig. 2(c) for illustration). However, none of the two planar bounds, including the one in [ 23 ] and the dual bound, is the optimal under all different situations. To refine the approximation, we need to devise new approaches that fuse the two planar bounds, meanwhile holding the linearity of the bounds to maintain verification efficiency. Contributions In this paper, we explore a novel usage of ReLU functions, by which we propose \mathsf{BuFFeT} , standing for B o U nd F using-based re F in E ment for T ransformers, which fuses the planar bounds of dot product operations and represents them with the assistance of ReLU functions. The advantage of doing this is that, we can then leverage the relaxed linear bounds of the ReLU-based representations, by exploiting the rich body of existing literature in classic neural network verification, in which linear relaxation of ReLUs has been extensively studied and a variety of bound-tightening techniques have been developed. Consequently, we can derive tighter bounds for dot products—the fundamental operations in self-attention layers—which in turn leads to tighter bounds on transformer outputs. Moreover, based on our representation, we also show that the bounds proposed in [ 23 ] and its dual case, are two specific instantiations that can be derived by our representation, and we explain why they are suboptimal. Furthermore, we leverage the insights from two existing strategies from classic neural network verification to derive improved approximation. The first strategy, adapted from [ 24 ] , is simple but effective, which, based on the input ranges of ReLUs, selects the optimal planar bound to refine the approximation. The second strategy, inspired by [ 32 ] , is

Read the complete text