CAMBench-QR : A Structure-Aware Benchmark for Post-Hoc Explanations with QR Understanding
Ritabrata Chakraborty, Avijit Dasgupta, Sandeep Chaurasia · 2025
Read CAMBench-QR : A Structure-Aware Benchmark for Post-Hoc Explanations with QR Understanding by Ritabrata Chakraborty, Avijit Dasgupta, Sandeep Chaurasia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16745.txt.
Introduction
Ritabrata Chakraborty, Avijit Dasgupta, and Sandeep Chaurasia test whether AI explanations point at the right spots in a 2025 paper. Heat maps can look right while missing the true structure. The team use QR codes as a test bed, since their layout is fixed and known. A good map should light the code's key parts and skip the background.
Their benchmark scores maps on that structural fit. It also ships with metrics and training recipes others can copy. The test acts as a quick check of whether a map is truly structure-aware. Such checks matter when explanations guide trust in medical or safety tools.
Catalog record
| CREATOR | Ritabrata Chakraborty, Avijit Dasgupta, Sandeep Chaurasia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-16745
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CAMBench-QR : A Structure-Aware Benchmark for Post-Hoc Explanations with QR Understanding Ritabrata Chakraborty, Avijit Dasgupta, Sandeep Chaurasia Abstract Visual explanations are often plausible but not structurally faithful. We introduce CAMBench-QR, a structure-aware benchmark that leverages the canonical geometry of QR codes (finder patterns, timing lines, module grid) to test whether CAM methods place saliency on requisite substructures while avoiding background. CAMBench-QR synthesizes QR/non-QR data with exact masks and controlled distortions, and reports structure-aware metrics (Finder/Timing Mass Ratios, Background Leakage, coverage AUCs, Distance-to-Structure) alongside causal occlusion, insertion/deletion faithfulness, robustness, and latency. We benchmark representative, efficient CAMs (LayerCAM, EigenGrad-CAM, XGrad-CAM) under two practical regimes of zero-shot and last-block fine-tuning. The benchmark, metrics, and training recipes provide a simple, reproducible yardstick for structure-aware evaluation of visual explanations. Hence we propose that CAMBENCH-QR can be used as a litmus test of whether visual explanations are truly structure-aware. 1 Introduction “Are you watching closely?” Alfred Borden, The Prestige The question is fitting for explainable vision: saliency maps can be compelling misdirection. On natural images, visually plausible heatmaps often look right without being correct . When the target class exhibits rigid geometry and canonical parts , explanations should be structurally faithful : attribution ought to concentrate on the parts that constitute the concept and avoid incidental texture that merely correlates with the label. Figure 1 contrasts these settings: a cat admits many aesthetically reasonable explanations, whereas a QR code has non-negotiable components—three finder patterns, timing lines, and a module grid—so the “where to look” is objectively knowable. Figure 1 : Plausible vs. structural explanations. (Top) On natural images, CAMs may look convincing without ground truth for “where to look.” (Bottom) For QR codes, canonical parts are known, allowing objective, structure-aware evaluation. Structural faithfulness matters because CAM explanations inform auditing, debugging, and deployment decisions. If evaluation rewards plausible masks instead of part-level alignment , models can appear trustworthy while relying on shortcut evidence (borders, backgrounds, repetitive textures). This risk is acute in high-frequency or layout-driven domains—codes, scientific imagery, documents—where texture and geometry are easily confounded: a classifier may leverage backgrounds, and a CAM may faithfully reflect that shortcut. A structure-aware perspective reframes the question from does the map look reasonable? to does the evidence land on the parts that make the concept work—and stay off everything else? QR codes offer an unusual opportunity to study this precisely. Their geometry is canonical and parameterizable, enabling synthesis of QR/non-QR data with exact part masks and controlled distortions. This permits measurements that standard pixel-overlap proxies cannot cleanly capture: how much saliency mass falls on finder and timing structures versus background; how the confident core covers required parts across thresholds; how far spurious mass drifts from structure; and whether structural mass predicts causal influence when parts are occluded or gradually inserted/deleted. Because the generation process is controlled, we can apply rotation, perspective, blur, JPEG compression, low light, and occlusion while preserving ground-truth structure. To interpret results beyond any one method, we organize CAMs along three orthogonal axes (Fig. 3 ): how pixel importance is computed (gradients, activations, perturbations), how feature maps are aggregated (global pooling, layer-wise fusion, subspace denoising), and computational profile (single-pass lightweight versus multi-mask/ablation). We further separate the explainer rule from the feature basis it aggregates: keeping the rule fixed while varying trainable depth changes the basis (last block versus deeper) without altering inference-time cost. This factorization lets us examine zero-shot use (frozen backbone + linear head), last-block fine-tuning, and a simple leakage-minimizing penalty in a controlled way. Research Questions RQ1 Do structure-aware metrics—mass on finder/timing modules, background leakage, coverage AUCs, and distance-to-structure—agree with causal tests such as part-occlusion correlations and insertion/deletion curves? RQ2 How does trainable depth (head-only \rightarrow last block \rightarrow deeper) reshape attribution structure when the explainer rule is held fixed? RQ3 Can a light penalty suppress off-structure mass without collapsing in-QR coverage? RQ4 How stable are structure-aware explanations under common distortions? Contributions. Our contributions can be summarized as follows: • A structure-aware dataset and protocol. Synthetic QR/non-QR corpora with exact finder, timing, and box masks plus controlled distortions, enabling geometry-aware evaluation beyond pixel-overlap proxies. • Metrics that target parts and avoid shortcuts. Mass-on-structure ratios (FMR/TMR), background leakage (BL), structure-coverage AUCs, and normalized distance-to-structure (DtS), complemented by targeted part-occlusion correlations, insertion/deletion faithfulness, and latency for deployability. • A factorized study across methods and regimes. A principled axes-based comparison of representative, efficient CAM families under two practical regimes—zero-shot and last-block fine-tuning—with an optional leakage-minimizing penalty; experiments span ResNet-50 [ 8 ] and ConvNeXt-B [ 12 ] . • Simple training recipes that improve structure without architectural changes. Short schedules that adapt only the last residual stage to re-orient high-level features toward canonical parts, and a lightweight penalty that suppresses off-structure mass, both with negligible inference-time overhead. The rest of the paper is structured as follows. Section 2 discusses related literature to CAM models, explanation evaluations and structure aware studies. Section 3 discusses the pipeline of CAMBENCH-QR, taxonomy of the axes of our study and how we use the proposed metrics for structural evaluation. This is consolidated by Section 4 which shows quantitative results for the benchmark. We discuss findings and takeaways in Section 5 and conclude the paper in Section 6 . 2 Related Works CAM and its modern variants. Class Activation Mapping (CAM) demonstrated that inserting global average pooling allowed models trained with image-level labels to still reveal spatially discriminative regions, sparking a large family of class-discriminative heatmaps [ 25 ] . Grad-CAM removed the architectural constraint of CAM by using gradients to generalize to arbitrary CNNs and tasks [ 18 ] . Ideas consequent to this centered around sharpening or stabilizing maps. For instance, Grad-CAM++ improved multi-instance localization and weighting [ 3 ] , while Smooth Grad-CAM++ smoothened activations for crisper maps [ 16 ] . A parallel, gradient-free branch weights features via forward signals: Score-CAM [ 21 ] , with refinements SS-CAM [ 20 ] and IS-CAM [ 15 ] . Ablation-CAM estimates causal importance by ablating feature maps [ 6 ] . To reduce query/runtime cost, Group-CAM aggregates and perturbs grouped activations [ 24 ] ; ReciproCAM proposes a lightweight, gradient-free variant via spatial perturbations [ 2 ] . Axiomatic work formalizes desiderata such as Sensitivity/Conservation in XGrad-CAM [ 7 ] . Finally, to capture detail across scales, Layer-CAM aggregates evidence from multiple layers [ 11 ] , while Eigen-CAM replaces gradients with a PCA projection of activations for object-focused maps [ 14 ] . CAM-style explanations for Transformers. CAM-style relevance propagation has been adapted to attention architectures. Chefer et al. derive class-specific relevancy for pure self-attention Transformers and extend to encoder–decoder and bimodal settings, enabling explanations across ViT/DETR/VQA-style models [ 5 , 4 ] . Formal evaluation of explanations. Early quantitative proxies emphasized localization. The Pointing Game, introduced with Excitation Backprop counts hits when the max-saliency lands inside the target region [ 23 ] . Sanity checks revealed methods that are insensitive to model parameters or data, emphasizing validity over aesthetics [ 1 ] . Causal faithfulness is probed by ROAR (remove top regions and retrain; larger accuracy drops imply more faithful attributions) [ 10 ] , and by black-box Insertion/Deletion curves via RISE [ 17 ] . Theoretically grounded criteria like