TriGuard: Testing Model Safety with Attribution Entropy, Verification, and Drift

Dipesh Tharu Mahato, Rohan Poudel, Pramod Dhungana · 2025

Read TriGuard: Testing Model Safety with Attribution Entropy, Verification, and Drift by Dipesh Tharu Mahato, Rohan Poudel, Pramod Dhungana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14217.txt.

Introduction

Dipesh Tharu Mahato, Rohan Poudel, and Pramod Dhungana present a three-part safety test for neural networks. Their 2025 paper starts from a known gap: high accuracy does not prove a model reasons in a stable way. Attacks and shifts in data can expose hidden weakness.

Their framework, TriGuard, joins formal checks of robustness with two explanation-based measures. One counts how focused a model's stated reasons are, and the other tracks how much those reasons drift under change. Tests across three data sets and five model types found verified models whose reasoning was still unstable. Training that rewards steadier explanations cut drift without hurting accuracy. The work argues that safety checks should grade reasons, not just answers.

Catalog record

CREATORDipesh Tharu Mahato, Rohan Poudel, Pramod Dhungana
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-14217

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TriGuard: Testing Model Safety with Attribution Entropy, Verification, and Drift Dipesh Tharu Mahato, Rohan Poudel, Pramod Dhungana Abstract Deep neural networks often achieve high accuracy, but ensuring their reliability under adversarial and distributional shifts remains a pressing challenge. We propose TriGuard 1 1 1 Project code: https://github.com/dipeshbabu/TriGuard , a unified safety evaluation framework that combines (1) formal robustness verification, (2) attribution entropy to quantify saliency concentration, and (3) a novel Attribution Drift Score measuring explanation stability. TriGuard reveals critical mismatches between model accuracy and interpretability: verified models can still exhibit unstable reasoning, and attribution-based signals provide complementary safety insights beyond adversarial accuracy. Extensive experiments across three datasets and five architectures show how TriGuard uncovers subtle fragilities in neural reasoning. We further demonstrate that entropy-regularized training reduces explanation drift without sacrificing performance. TriGuard advances the frontier in robust, interpretable model evaluation. 1 Introduction Deep learning has driven transformative advances in computer vision, enabling convolutional neural networks (CNNs) and their variants to achieve state-of-the-art performance across image classification, detection, and segmentation tasks. Yet despite their predictive power, modern neural models remain brittle in the face of adversarial examples ( Goodfellow et al. 2015 ) , distributional shifts ( Hendrycks et al. 2021 ) , and perturbation-based attacks ( Croce & Hein 2020 ) . These vulnerabilities pose significant concerns in safety-critical applications such as autonomous driving, medical imaging, and financial screening, where model decisions must be both accurate and trustworthy. The central problem is that accuracy alone is not a sufficient metric for reliable deployment. A model may achieve high test accuracy yet fail dramatically when exposed to slight perturbations, either crafted adversarially or arising from natural variation. This has led to a growing emphasis on robustness evaluation and explainability as core components of model assessment ( Hooker et al. 2020 ; Xu et al. 2020 ) . Robustness and Attribution as Safety Axes. Two main axes have emerged as vital to robust and interpretable models: (1) formal verification of robustness, which guarantees prediction stability under small input changes; and (2) attribution-based interpretability, which assesses the saliency and stability of model explanations. Each axes offers partial insight: formal verification answers “Will the prediction change under bounded perturbations?” while attribution asks “What parts of the input most influenced this decision?” However, these have rarely been integrated into a unified diagnostic framework. Recent work has attempted to measure robustness using tools like DeepPoly ( Singh et al. 2019 ) , CROWN-IBP ( Zhang et al. 2022 ) , and ERAN ( Katz et al. 2021 ) , offering formal guarantees but often limited to small networks or idealized settings. Parallelly, attribution methods such as Integrated Gradients ( Sundararajan et al. 2017 ) , SmoothGrad ( Smilkov et al. 2017 ) , and saliency maps ( Simonyan et al. 2014 ) have become popular in model interpretation. However, these methods suffer from instability: attributions may change drastically under imperceptible input shifts or minor weight changes, undermining trust in their explanations ( Kindermans et al. 2019 ) . Even more concerning, attribution instability has been shown to correlate with model susceptibility to adversarial examples ( Ghorbani et al. 2019 ) . Recent efforts like ROAR ( Hooker et al. 2020 ) and AOPC ( Samek et al. 2017 ) suggest that measuring attribution reliability not just saliency is crucial. Yet, a comprehensive framework that triangulates prediction correctness, robustness guarantees, and attribution stability remains missing. Our Contribution: TriGuard. To address this gap, we propose TriGuard, a scalable evaluation framework that triangulates three complementary safety axes: 1. Formal Verification : We use adversarially bounded checks (PGD+interval bound propagation) to assess whether predictions remain invariant within an \epsilon -ball in input space. 2. Attribution Entropy : We quantify the focus of explanations by measuring entropy over normalized Integrated Gradients. Lower entropy indicates sparse, localized attention hypothesized to reflect more interpretable reasoning. 3. Contrastive Attribution Drift : We introduce a new metric Attribution Drift Score which captures how model explanations change between two neighboring baseline inputs. Large drifts may signal sensitivity to spurious features or reliance on unstable gradients. 2 Methods In this section, we describe the components of the TriGuard framework for evaluating neural network safety. TriGuard is designed around three diagnostic axes: formal robustness verification, attribution-based entropy analysis, and contrastive attribution drift. We also describe the experimental protocols used to implement and validate each axes across multiple datasets and model architectures. 2.1 Model and Dataset Setup Let f_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{k} be a deep neural network classifier parameterized by weights \theta , where d is the input dimension and k is the number of output classes. Given an input x\in\mathbb{R}^{d} , the predicted class is: \hat{y}=\arg\max_{j}f_{\theta}(x)_{j} We evaluate this model across standard vision datasets: • MNIST (gray-scale digits), • FashionMNIST (gray-scale clothing images), • CIFAR-10 (RGB natural images). Each model is trained for 5 epochs using Adam, and tested on both clean inputs and adversarially perturbed examples. 2.2 Axis 1: Formal Verification of Robustness To evaluate whether model predictions remain stable under small perturbations, we define an \epsilon -bounded region around an input x as: B_{\epsilon}(x)=\left\{x^{\prime}\in\mathbb{R}^{d}:\|x^{\prime}-x\|_{\infty}\leq\epsilon\right\} A model is said to be formally robust at x if: \forall x^{\prime}\in B_{\epsilon}(x):\arg\max f_{\theta}(x^{\prime})=\hat{y} We implement a bounded adversarial check using Projected Gradient Descent (PGD) to search for a counterexample within B_{\epsilon}(x) . If no such x^{\prime} is found after T steps of perturbation with step size \alpha , we consider the model to be locally robust : x_adv = x + alpha * sign(grad) x_adv = clip(x_adv, x - eps, x + eps) where sign(grad) is the sign of the gradient of the loss with respect to the input. We also use a simplified Interval Bound Propagation (IBP) style check to ensure robustness holds across all \epsilon -perturbed inputs using forward bounds, implemented as: x_{\min}=x-\epsilon,\quad x_{\max}=x+\epsilon The model passes the check if the predicted class dominates across all bounds. 2.3 Axis 2: Attribution Entropy Given a model and input x , we compute Integrated Gradients (IG) as the attribution method. IG for the i -th input feature is defined as: \text{IG}_{i}(x)=(x_{i}-x_{i}^{\prime})\cdot\int_{\alpha=0}^{1}\frac{\partial f_{\theta}(x^{\prime}+\alpha(x-x^{\prime}))}{\partial x_{i}}\,d\alpha where x^{\prime} is a baseline input (typically a zero vector or an average image). To compute this in practice, we use a Riemann approximation with m steps: \text{IG}_{i}(x)\approx(x_{i}-x_{i}^{\prime})\cdot\frac{1}{m}\sum_{j=1}^{m}\frac{\partial f_{\theta}\left(x^{\prime}+\frac{j}{m}(x-x^{\prime})\right)}{\partial x_{i}} We normalize the attributions to obtain a probability distribution p\in\Delta^{d} , and compute its entropy: H(p)=-\sum_{i=1}^{d}p_{i}\log(p_{i}+\delta) where p_{i}=\frac{|\text{IG}_{i}|}{\sum_{j}|\text{IG}_{j}|} and \delta is a small constant added to avoid numerical instability from \log(0) . Low entropy H(p) implies that the attribution is concentrated (potentially more interpretable), whereas high entropy suggests that the attribution is diffuse or noisy. 2.4 Axis 3: Contrastive Attribution Drift We define the Attribution Drift Score (ADS) to quantify the shift in saliency maps when comparing two semantically equivalent inputs. For a fixed test input x , we compute its attribution vectors a^{(1)},a^{(2)} using Integrated Gradients (IG) under two different baselines x^{(1)},x^{(2)} , and define: \text{ADS}(x^{(1)},x^{(2)})=\|a^{(1)}-a^{(2)}\|_{2} This score reflects how sensitive a model’s explanation is to the choice of attribution reference — serving as a proxy for explanation stability under interpretability perturbations. In our experiments, we use two commonly accepted and semantically meaningful baselines: • A zero baseline (all pixels set to zero), which represents complete absence of input

Read the complete text