Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers
Nikita Kezins, Urbas Ekka, Pascal Berrang, Luca Arnaboldi · 2026
Read Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers by Nikita Kezins, Urbas Ekka, Pascal Berrang, Luca Arnaboldi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10901.txt.
Introduction
Nikita Kezins, Urbas Ekka, Pascal Berrang, and Luca Arnaboldi move beyond red-teaming for language-model guardrails, seeking formal proof that a classifier blocks harmful prompts. Testing alone gives no guarantees, and harm has no neat definition in raw input space, so standard robustness proofs do not transfer.
Their fix verifies in the classifier's pre-activation space instead, enclosing known harmful prompts in a convex region and certifying its worst point, which suffices by the monotonicity of the sigmoid head. Two region constructions yield exact or probabilistic certificates. Applied to three classifiers on toxicity, every exact check exposed verifiable safety holes. Safety teams get proofs of failure, and a path toward proofs of safety.
Catalog record
| CREATOR | Nikita Kezins, Urbas Ekka, Pascal Berrang, Luca Arnaboldi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46194 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10901
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers Nikita Kezins, Urbas Ekka, Pascal Berrang, Luca Arnaboldi Abstract Guardrail Classifiers defend production language models against harmful behavior, however, although results may seem promising in testing, they provide no formal guarantees. Unfortunately, providing formal guarantees for such models is hard because “harmful behavior” has no natural specification in a discrete input space: and the standard ϵ \epsilon -ball properties used in other domains do not carry semantic meaning. We close this gap by shifting verification from the discrete input space to the classifier’s pre-activation space, where we define a harmful region as a convex shape enclosing the representations of known harmful prompts. Because the sigmoid classification head is monotonic, certifying the worst-case point of this region is sufficient to certify the entire region, yielding a closed-form soundness proof without approximation in O ( d ) O(d) time. To formally evaluate these types of classifiers, we propose two constructions of such regions: SVD-aligned hyper-rectangles, which yield exact SAT/UNSAT certificates, and Gaussian Mixture Models, which yield probabilistic certificates over semantically coherent clusters. Applying this framework to three author-trained Guardrail Classifiers (BERT, GPT-2 and Llama-3.1-8B) on the toxicity domain, every hyper-rectangle configuration returns SAT, exposing verifiable safety holes across all classifiers, despite seemingly high empirical metrics (F1, recall, etc). Probabilistic GMM certificates also expose a divergent structural stability in how these models represent harm. While GPT-2 and Llama-3.1-8B maintain robust coverage of 90% and 80% across varying boundaries, BERT’s safety guarantees prove uniquely volatile. This ‘coverage collapse’ to 55% at the optimal threshold ( τ ∗ \tau^{*} ) reveals a sparsely populated safety margin in BERT, which only achieves full coverage by adopting an extremely conservative pessimistic threshold. These approaches combined, provide new insights on how effective Guardrail Classifiers really are, beyond traditional red-teaming. 1 Introduction The rapid integration of Large Language Models (LLMs) into production environments has intensified the need for robust safety guardrails. Among the most prominent defenses are Guardrail Classifiers – an umbrella term encompassing rule-based systems 14 , input-output safeguards 8 , and state-of-the-art Constitutional Classifiers 15 ; 3 . These models serve as specialized safety filters to detect and intercept adversarial jailbreaks. These classifiers demonstrate strong empirical resilience against complex multi-turn attacks and universal adversarial triggers. However, their safety claims rest exclusively on red-teaming results, and red-teaming establishes only the absence of attacks tried, not the absence of attacks possible. Formal verification offers a complementary path by providing mathematical guarantees over entire input regions. Applying it to Transformer-based classifiers, however, runs into two well-known obstacles. First, standard verification techniques such as Satisfiability Modulo Theories (SMT) or linear relaxation-based methods 9 methods do not scale to the depth and non-linearity of modern Transformers; as depth grows, approximation errors accumulate and the resulting bounds become too loose to be informative 6 ; 12 . Second, the discrete nature of language offers no analogue of the L_{\infty} -balls used in computer vision: a tractable, semantically meaningful neighbourhood in token space remains an open problem. Together, these obstacles have left a clear gap – no prior work has formally verified whether a Guardrail Classifier actually enforces its constitution. We close this gap with the first formal safety analysis of Guardrail Classifiers, replacing attack-success rates with mathematically provable certificates. Our approach sidesteps both obstacles by relocating verification. Rather than propagating bounds through the full Transformer, we verify only the classification head on the last hidden state. Rather than defining neighbourhoods in token space, we define safety regions directly in this representation space, where semantically similar inputs cluster geometrically. This approach leverages a key architectural insight: the monotonicity of the sigmoid activation function, which permits exact, closed-form analysis of whether a given safety region lies entirely above the classifier’s threshold – UNSAT, the region is classified correctly – or admits the existence of a counterexample – SAT, the classifier has a safety hole within the specification. This formulation lets us pose three questions about Guardrail Classifiers in a falsifiable way. RQ1: Do Guardrail Classifiers trained on realistic safety data exhibit verifiable safety holes over semantic regions of harmful behavior? RQ2: How does specification fidelity and certified coverage vary across classifiers and model scales? RQ3: What are the trade-offs between deterministic (hyper-rectangles) and probabilistic (GMM) specifications, and how does threshold selection affect the strength of the resulting certificate? Our contributions are three-fold: A closed-form gap-analysis framework for Guardrail Classifiers: We prove that, for classifiers with a sigmoid head, deciding whether a region bounded by a hyper-rectangle lies entirely above the classification threshold reduces to evaluating the head at a single worst-case point. The complexity of the procedure is linear for the verification of each hyper-rectangle ( O(d) ). Two complementary specifications of harm: We adapt the hyper-rectangle approach introduced in ANTONIO 2 for deterministic SAT/UNSAT certificates and introduce a Gaussian Mixture Model construction for probabilistic certificates over semantically coherent clusters of harm. Empirical safety gap analysis across classifiers: Applying our framework to three author-trained Guardrail Classifiers (BERT, GPT-2 and Llama-3.1-8B) on the toxicity domain, every hyper-rectangle configuration returns SAT, confirming safety holes across all classifiers. GMM certificates reveal substantial variation in certified coverage – GPT-2 (90%), Llama-3.1-8B (80%) and BERT (55%) under the optimal threshold \tau^{*} – with BERT reaching 100% under the pessimistic threshold \tau_{pess} . 1 1 1 Source code is available at https://github.com/entfane/formal_verification_guardrails . 2 Formal Preliminaries We now formalize the mathematical conditions required to certify a Guardrail Classifier, establishing the theoretical foundation for the two verification methods developed in Section 3 . We consider a fixed Transformer backbone \Phi:\mathcal{X}\to\mathbb{R}^{d} and a classification head f:\mathbb{R}^{d}\to(0,1) , where f(\mathbf{x})=\sigma(W^{\top}\mathbf{x}+b) , W\in\mathbb{R}^{d} are the head weights, b\in\mathbb{R} is the bias, and \sigma is the sigmoid activation. The core challenge is to verify that for a semantic region \mathcal{R}\subset\mathbb{R}^{d} in the pre-classification activation space, the property f(\mathbf{x})>\tau holds \forall\mathbf{x}\in\mathcal{R} . Exact Bounds via Sigmoid Monotonicity. The primary advantage of verifying the classification head in isolation is the exploitation of its monotonic properties. Since \sigma is strictly increasing, minimizing f over \mathcal{R} reduces to minimizing the pre-activation z=W^{\top}\mathbf{x}+b over \mathcal{R} . In the case \mathcal{R} is a hyper-rectangle \mathcal{H} defined by lower bounds \mathbf{l} and upper bounds \mathbf{u} , the minimum pre-activation value z_{\min} is given by: z_{\min}=\sum_{i=1}^{d}\min(w_{i}l_{i},w_{i}u_{i})+b (1) Because \sigma(z) is strictly increasing, the global minimum of f over \mathcal{H} is \sigma(z_{\min}) . A region \mathcal{R} is certified as formally safe if \sigma(z_{\min})>\tau . This closed-form solution bypasses iterative solvers or SMT-based search entirely. The formal proof is provided in Appendix B . Probabilistic Bounds via Gaussian Transformations. For probabilistic verification, because the classification head applies a linear transformation before the sigmoid, each component k induces a univariate Gaussian over the pre-activation z : z\sim\mathcal{N}(W^{\top}\mu_{c}+b,\;W^{\top}\Sigma_{c}W) (2) Rather than propagating distributions through the non-linear sigmoid, we map the threshold \tau back into pre-activation space via the inverse sigmoid \sigma^{-1}(\tau)=\ln\!\left(\frac{\tau}{1-\tau}\right) and compute the probability mass exceeding it: P(z>\sigma^{-1}(\tau))=1-\Phi\!\left(\frac{\sigma^{-1}(\tau)-\mu_{z}}{\sigma_{z}}\right) (3) where \Phi is the standard normal CDF. This yields a probabilistic safety certificate: the probability that a harmful activation drawn from the modelled distribution is correctly classified above the threshold. 3 Method We verify harmful regions defined within the last hidden state of the Transformer, immediately preceding the classification