B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data

Magdalena Proszewska, Tomasz Danel, Dawid Rymarczyk · 2025

Read B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data by Magdalena Proszewska, Tomasz Danel, Dawid Rymarczyk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22252.txt.

Introduction

Magdalena Proszewska, Tomasz Danel, and Dawid Rymarczyk build a test set in a 2025 paper on explaining AI for chemistry. Graph-based models predict traits of molecules, and chemists need to know why a model made its call. Past tests of explanation tools used fake data or simple tasks, so they missed real-world strain.

Their benchmark, B-XAIC, is built from real molecule data with tasks whose true reasons are known. That design lets each explanation be checked against the actual basis for a label. A broad run of current tools on the set exposed clear limits in the molecule domain. The authors offer the set as a shared yardstick. Better scores on it should mean explanations chemists can trust.

Catalog record

CREATORMagdalena Proszewska, Tomasz Danel, Dawid Rymarczyk
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22252

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data Magdalena Proszewska, Tomasz Danel, Dawid Rymarczyk Abstract Understanding the reasoning behind deep learning model predictions is crucial in cheminformatics and drug discovery, where molecular design determines their properties. However, current evaluation frameworks for Explainable AI (XAI) in this domain often rely on artificial datasets or simplified tasks, employing data-derived metrics that fail to capture the complexity of real-world scenarios and lack a direct link to explanation faithfulness. To address this, we introduce B-XAIC, a novel benchmark constructed from real-world molecular data and diverse tasks with known ground-truth rationales for assigned labels. Through a comprehensive evaluation using B-XAIC, we reveal limitations of existing XAI methods for Graph Neural Networks (GNNs) in the molecular domain. This benchmark provides a valuable resource for gaining deeper insights into the faithfulness of XAI, facilitating the development of more reliable and interpretable models. 1 Introduction Graph Neural Networks (GNNs) have become the standard for predictive modeling of small molecules ( 40 ) , achieving exceptional performance across property prediction, virtual screening, and related pharmaceutical tasks. While their predictive capabilities are well-established, a growing emphasis is now placed on understanding their reasoning ( 16 ) . In scientific applications of deep learning to small molecules, transparent explanation mechanisms are not merely desirable but crucial. They build researcher trust, ensure model reliability, and potentially uncover novel insights that can accelerate drug discovery and materials design ( 43 ) . To address this gap, a range of Explainable AI (XAI) techniques have been adapted or specifically designed for GNNs, aiming to reveal the mechanisms behind their predictions ( 16 , 17 ) . These approaches generally fall into two categories: counterfactual methods ( 8 , 21 , 37 ) , which seek to identify minimal input changes that alter a model’s prediction, and factual methods ( 22 , 31 , 45 ) , which aim to highlight important substructures within the input graph. Factual methods further diverge into post-hoc explainers ( 45 ) , that analyze a trained black-box model, and inherently interpretable architectures [ 14 , 38 , 50 ] , which aim for transparency through their design. However, recent findings indicate a critical challenge: regardless of the specific XAI method used, the resulting explanations can be unreliable, or even misleading, potentially interfering with scientific understanding [ 13 ] . Despite the strong performance of these models on established benchmarks and metrics for GNNs and small molecules, this is still observable. In response to the limitations of current XAI evaluation for GNNs, the community has developed synthetic datasets ( 2 , 4 , 22 , 42 , 45 ) . However, these often lack real-world complexity, while creating real-world datasets with ground truth explanations is challenging or impossible. Existing real-world datasets like MUTAG are small and task-limited. Furthermore, many evaluation methods rely on thresholding importance maps or selecting top-k elements, which can be problematic for tasks dependent on the presence/absence of substructures, where no single element is inherently more important. This arbitrary selection can yield inaccurate explanations and misleading metrics. While AUROC ( 6 , 49 ) avoids thresholding, it becomes ineffective when no specific element is important, leading to empty ground truth explanations. To address these limitations, we introduce B-XAIC (Benchmark for eXplainable Artificial Intelligence in Chemistry), a novel benchmark comprising 50K small molecules and 7 diverse tasks, accompanied by both ground truth labels and corresponding explanations, making accuracy-based metrics a directly applicable and reliable evaluation method. B-XAIC tackles the challenges associated with thresholding explanations or selecting the top-k most important elements by considering two distinct scenarios: (1) cases where a specific part of the input graph constitutes the explanation, which can be effectively evaluated using AUROC or Average Precision (AP), and (2) cases where the entire graph is equally important for the prediction, in which the evaluation focuses on ensuring the explanation does not contain irrelevant outliers. Ultimately, B-XAIC enables a direct and fair comparison of various factual XAI approaches, both post-hoc explainers and inherently self-explainable models. 2 Related Work Explanability in GNNs. Recent research in Graph Neural Networks (GNNs) has increasingly focused on developing methods to interpret and explain the decisions made by these models. Explainable AI (XAI) techniques for GNNs can be broadly categorized based on the type of explanation they provide ( 17 ) . These methods may involve identifying key substructures within the input graph that influence the model’s predictions, offering factual explanations by highlighting relevant parts of the input ( 10 , 22 , 45 , 48 ) , or generating counterfactual examples where the input is perturbed in such a way that it leads to a different prediction outcome ( 8 , 21 , 37 ) . Furthermore, factual methods for explaining GNN predictions can be broadly classified into post-hoc and self-interpretable approaches. Post-hoc methods aim to explain the predictions of a pre-trained GNN by identifying important nodes, edges, or features that influence the model’s decision ( 22 , 31 , 45 ) . In contrast, self-interpretable methods design the GNN architecture to inherently incorporate explainability using information constraints, such as attention blocks ( 27 , 38 ) or bottlenecks ( 41 ) , or integrating structural constraints like prototypes ( 29 , 50 ) or graph kernels ( 9 , 14 ) , to ensure that the model is more interpretable by design. Explainability Benchmarks. The need for appropriate datasets to evaluate GNN explainability techniques has led to the introduction of various benchmark datasets with ground-truth explanations. Several synthetic datasets have been developed for node classification and graph classification tasks, where specific motifs serve as the ground truth. For instance, datasets like BA-Shapes, BA-Community, Tree Cycle, and Tree Grids ( 45 ) are designed for node classification, with the task of predicting whether a node is part of a known motif (such as a cycle, house, or grid). Similarly, synthetic datasets like BA-2Motifs ( 22 ) , BAMultiShapes ( 4 ) and Spurious Motifs ( 42 ) are designed for graph classification tasks, where the goal is to detect presence of given motifs in the entire graph. ShapeGGen ( 2 ) is a more recent development in the field of graph benchmarks. It is a synthetic graph generator designed to create a variety of graph datasets with diverse characteristics. While ShapeGGen provides valuable synthetic data, it remains limited by its artificial nature, which may not fully capture the complexity and noise present in real-world datasets. In addition to synthetic datasets, real-world datasets have been crucial for testing GNN explainability methods. Molecular datasets are particularly valuable in this context, as they can provide ground-truth explanations based on known chemical properties. Examples of such datasets include MUTAG ( 11 ) , Benzene, Fluoride-Carbonyl, and Alkane-Carbonyl ( 26 , 30 ) , which are graph classification tasks where explanations are based on the presence or absence of simple chemical structures: the NO 2 group in MUTAG, the benzene ring in Benzene, fluoride (F-) and carbonyl (C=O) groups in Fluoride-Carbonyl, and alkane (F-) and carbonyl (C=O) groups in Alkane-Carbonyl. These datasets contain 1.8K, 12K, 8.7K and 1.1K graphs, respectively. Although simple, these datasets serve as effective benchmarks and are commonly used

Read the complete text