DocVCE: Diffusion-based Visual Counterfactual Explanations for Document Image Classification
Saifullah Saifullah(✉), Stefan Agne, Andreas Dengel, Sheraz Ahmed · 2025
Read DocVCE: Diffusion-based Visual Counterfactual Explanations for Document Image Classification by Saifullah Saifullah(✉), Stefan Agne, Andreas Dengel, Sheraz Ahmed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04233.txt.
Introduction
Saifullah Saifullah, Stefan Agne, Andreas Dengel, and Sheraz Ahmed present DocVCE, a 2025 method for explaining document classifiers with pictures. Offices lean on models that sort scanned documents, yet heat maps of their reasons are hard to read and hide what the model learned overall. Their system instead builds a counter picture: the smallest edit that would flip the verdict.
A guided diffusion model first drafts a plausible flipped document in its hidden space. Then a patch-by-patch polish pulls it as close as possible to the original sheet. Across three document sets and three classifier kinds, the method scored well on the standard checks for such explanations. The authors pitch the counter pictures as actionable insight for high-stakes sorting.
Catalog record
| CREATOR | Saifullah Saifullah(✉), Stefan Agne, Andreas Dengel, Sheraz Ahmed |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-04233
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DocVCE: Diffusion-based Visual Counterfactual Explanations for Document Image Classification Saifullah Saifullah(✉), Stefan Agne, Andreas Dengel, Sheraz Ahmed Abstract As black-box AI-driven decision-making systems become increasingly widespread in modern document processing workflows, improving their transparency and reliability has become critical, especially in high-stakes applications where biases or spurious correlations in decision-making could lead to serious consequences. One vital component often found in such document processing workflows is document image classification, which, despite its widespread use, remains difficult to explain. While some recent works have attempted to explain the decisions of document image classification models through feature-importance maps, these maps are often difficult to interpret and fail to provide insights into the global features learned by the model. In this paper, we aim to bridge this research gap by introducing generative document counterfactuals that provide meaningful insights into the model’s decision-making through actionable explanations. In particular, we propose DocVCE, a novel approach that leverages latent diffusion models in combination with classifier guidance to first generate plausible in-distribution visual counterfactual explanations, and then performs hierarchical patch-wise refinement to search for a refined counterfactual that is closest to the target factual image. We demonstrate the effectiveness of our approach through a rigorous qualitative and quantitative assessment on 3 different document classification datasets—RVL-CDIP, Tobacco3482, and DocLayNet—and 3 different models—ResNet, ConvNeXt, and DiT—using well-established evaluation criteria such as validity, closeness, and realism. To the best of the authors’ knowledge, this is the first work to explore generative counterfactual explanations in document image analysis. The source code for this work will be made publicly available upon acceptance. 1 Introduction As black-box deep learning (DL)-based information extraction models continue to achieve remarkable performance breakthroughs across a wide variety of document analysis tasks [ 41 , 22 , 45 , 35 ] , a primary concern that still hinders their full integration into modern automated workflows—especially in high-stakes scenarios such as legal or law enforcement applications—is their inherent lack of transparency [ 56 , 39 , 57 ] and reliability [ 4 , 18 , 40 ] . This becomes especially relevant in light of recent regulations, such as the GDPR [ 9 ] and the AI Act 2022, which impose legal requirements on these systems to be both transparent and accountable for their decisions. Several recent studies have shown that most black-box deep learning models are not only susceptible to making unfair and biased decisions [ 32 , 12 , 29 , 21 , 39 ] —typically stemming from hidden biases in the training data—but also exhibit vulnerabilities when faced with out-of-distribution data at test time [ 4 , 18 , 40 ] . For instance, a recent work by Saifullah et al. [ 42 ] highlighted the spurious correlations learned by the black-box DL models on the document classification task. Similarly, recently introduced document robustness benchmarks, such as RVL-CDIP-D [ 40 ] or RoDLA [ 7 ] have demonstrated the vulnerability of state-of-the-art document image analysis models to minor input data perturbations. Such failure modes have given rise to the field of eXplainable AI (XAI) which aims to make these models more transparent and accountable. A wide variety of explainability methods [ 5 , 36 ] have been proposed in recent years to interpret and validate the decisions and behaviors of deep learning models [ 37 , 44 , 49 , 30 , 48 , 31 ] , the majority of which can be broadly classified into two main categories: the attribution-based methods [ 37 , 44 , 49 , 30 , 48 ] and the generative counterfactual approaches [ 31 , 23 ] . While several recent works have extended attribution-based methods to various document analysis tasks [ 42 , 3 , 34 ] with promising results, the explanations produced by these methods are often difficult to interpret and fail to provide additional information on whether the model’s decisions are influenced by global typographical and structural characteristics, such as font styles, text formatting, or document layout. On the other hand, generative counterfactual explanations [ 10 , 2 , 23 ] have recently shown great promise in uncovering the influence of such abstract features in natural image classification. In particular, diffusion models [ 8 ] have recently gained considerable attention for generating counterfactual explanations [ 10 , 2 , 23 ] due to their high-quality generative capabilities. However, while diffusion has also been widely explored in recent years for synthetic document image generation [ 55 , 15 , 14 ] , we found no existing research that specifically focuses on diffusion-based counterfactual explanations for document image analysis. In this work, we explore the potential of generative visual counterfactual explanations for interpreting the decisions of deep-learning based document image classification models. In particular, we introduce diffusion-based visual counterfactual explanations for document image classification (DocVCE), a novel approach that first uses latent diffusion models [ 38 ] in combination with classifier-guided stochastic sampling [ 20 , 8 ] to generate high-level, in-distribution counterfactual images, and then performs hierarchical patch-wise refinement to search for a counterfactual that is closest to the target factual image. Finally, to further improve interpretability, a difference map is generated to highlight the regions of the image that were modified to create the counterfactual image. Overall, our contributions are two-fold: – We introduce DocVCE, a diffusion-based visual counterfactual explanation framework for document images that is capable of generating actionable counterfactual explanations for the task of document image classification. To the best of the authors’ knowledge, this is the first work to explore diffusion-based counterfactual explanations for document image analysis. – We present a comprehensive qualitative and quantitative evaluation of our proposed approach on a diverse set of document benchmark datasets: RVL-CDIP [ 16 ] , Tobacco3482, and DocLayNet [ 33 ] . Furthermore, using several well-established quantitative metrics, we demonstrate that our approach outperforms existing state-of-the-art approach LDCE [ 10 ] in terms of validity, closeness, and realism. 2 Related Work 2.1 eXplainable AI (XAI) The field of eXplainable Artificial Intelligence (XAI) has emerged as a prominent area of research [ 5 , 36 ] , with numerous methods developed for explaining the decisions of machine learning models [ 52 , 30 , 37 , 24 , 23 ] . These methods can be broadly categorized into three primary types: feature-attribution methods [ 48 , 47 , 52 , 30 , 37 ] , concept-based explanations [ 24 ] , and counterfactual explanations [ 26 , 23 ] . Feature-attribution methods Feature-attribution methods explain a model’s decision-making by assigning importance to the input (or intermediate) features and are particularly valued for their post-hoc applicability. These methods can be further divided into two main types: gradient-based attribution [ 48 , 44 , 52 , 46 ] and perturbation-based attribution [ 58 , 37 , 30 ] . Gradient-based methods [ 48 , 44 , 52 , 46 ] estimate the importance of features by computing the gradient of the model’s prediction with respect to the input. On the other hand, perturbation-based methods [ 58 , 37 , 30 ] perturb or ablate the input (or intermediate) features to assess their relevance for the model’s decision, either by directly measuring the model’s confidence scores [ 58 ] or