Technical Report — Extended Empirical Validation of the ESS
Explainability Solution Space: · 2026
Read Technical Report — Extended Empirical Validation of the ESS by Explainability Solution Space: (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01235.txt.
Introduction
Mestre and three colleagues extend checks of ESS beyond staff-loss forecasts to a mixed urban resource sharing system. The second case mixes tabular, time, and place data under multi-interest-group rules, with scored ranks of typical clear-AI groups in both contexts. Results show ESS rankings adapt in step with rule roles, risk shapes, and group setups. They are not domain-set, backing ESS as a general choice-support tool for clear-AI plans.
Catalog record
| CREATOR | Explainability Solution Space: |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29496 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.01235
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Technical Report — Extended Empirical Validation of the ESS Explainability Solution Space: Abstract Abstract. This technical report provides an extended empirical validation of the Explainability Solution Space (ESS) by applying its operationalisation methodology to a real-time bank fraud detection system—a domain that differs substantially from the employee attrition case study presented in the companion paper. The system under analysis is an XGBoost gradient-boosted ensemble deployed at a European retail bank to classify card transactions as fraudulent or legitimate within a 200 ms latency budget. Operating under regulatory substitution conditions—governed by PSD2, GDPR Article 22, and EBA reporting guidelines—the use case demands high-fidelity compliance documentation, actionable user recourse, and developer-grade debugging capacity. We instantiate the ESS with five representative XAI families (SHAP, LIME, Counterfactual Explanations, Surrogate/Rule Extraction, and Prototypes), compute the full property-to-stakeholder aggregation pipeline with substitution contextual multipliers, derive resource-aware multi-objective utility scores, and formulate a principled hybrid recommendation. Results confirm that the ESS generates differentiated, stable, and operationally meaningful technique rankings in this new domain, thereby supporting the generalisability of the framework beyond the HR tabular context. Keywords: Explainable Artificial Intelligence, Fraud Detection, Trustworthy AI, Regulatory Compliance, Requirements Engineering, Hybrid Intelligence, XGBoost, SHAP, Counterfactual Explanations 1. Introduction and Motivation Financial fraud constitutes one of the most significant operational risks confronting the retail banking sector. Card fraud losses in the European Union alone exceeded €1.8 billion annually as of the most recent reporting periods, with the proliferation of digital payment channels introducing novel attack vectors that challenge traditional rule-based detection systems ( European Banking Authority 2022 ) . As gradient-boosted ensemble models and deep learning architectures increasingly displace heuristic approaches, the opacity of these systems has come under intensified scrutiny from three distinct directions: regulatory bodies mandating the auditability of automated decisions under the Payment Services Directive 2 (PSD2) and GDPR Article 22 ( European Parliament 2016 ) ; operational analysts requiring comprehensible, action-oriented alerts when reviewing flagged transactions or responding to cardholder disputes; and machine learning engineers demanding robust debugging and monitoring mechanisms to manage model drift in adversarial, non-stationary data environments. This technical report applies the Explainability Solution Space (ESS) ( Mestre et al. 2026 ) to a representative real-time fraud detection system deployed at a European retail bank. The ESS, introduced in the companion paper, is a three-dimensional operational framework that positions XAI techniques along the axes of Compliance ( C ), User comprehensibility ( U ), and Developer utility ( D ), enabling systematic, reproducible evaluation of explanation strategies through intrinsic property vectors, stakeholder-weighted aggregation functions, and contextual multipliers. The primary objectives of this report are twofold: (i) to demonstrate the applicability and consistency of the ESS methodology in a domain characterised by hard real-time constraints, regulatory substitution conditions, and a dual user population; and (ii) to produce a structured, citable analysis that complements the HR attrition instantiation of the companion paper, thereby contributing to the empirical generalisation of the framework. The fraud detection context introduces several challenges not present in the HR setting. Decisions are made at millisecond timescales on highly imbalanced datasets ( \sim 0.08% fraud prevalence), with direct financial and legal consequences for both cardholders and the issuing institution. Explanations must concurrently satisfy the needs of compliance auditors requiring tamper-evident audit logs, customer service agents translating model decisions into cardholder communication, and data scientists monitoring feature importance stability over production data streams. This multiplicity of heterogeneous stakeholder requirements renders fraud detection an ideal stress-test for the ESS. The remainder of this report is structured as follows. Section 2 describes the use case, system architecture, stakeholder analysis, and regulatory context. Section 3 operationalises the ESS pipeline—from property vector assignment through stakeholder-weighted aggregation and contextual adjustment to qualitative discretisation. Section 4 presents the resource-aware multi-objective selection analysis. Section 5 articulates the hybrid explainability recommendation derived from the ESS. Section 6 discusses domain-specific considerations, consistency with the companion paper, and limitations. Section 7 concludes. 2. Use Case Description 2.1 System Architecture The system under analysis is an XGBoost gradient-boosted ensemble classifier ( Chen and Guestrin 2016 ) deployed by a European retail bank to score individual card payment transactions in real time. Each incoming transaction is evaluated against the model within a hard latency budget of 200 ms (end-to-end, including feature engineering, model inference, and explanation generation). Transactions whose fraud probability exceeds a configurable operational threshold are either blocked automatically or routed to a human analyst review queue, depending on the confidence margin and transaction value. The system processes approximately 4.2 million transactions per day, of which roughly 0.08% are ultimately confirmed as fraudulent—a severe class imbalance representative of production fraud environments. The model is trained on 87 engineered features derived from four primary signal categories: (i) transaction attributes (amount, currency, merchant category code, point-of-sale channel); (ii) geospatial signals (delta between transaction location and cardholder home region, cross-border indicator); (iii) device and authentication signals (device fingerprint match, strong-customer-authentication method); and (iv) behavioural velocity features (transaction count and cumulative spend over 1-hour, 24-hour, and 7-day rolling windows relative to per-cardholder historical baselines). The model achieves an AUC-ROC of 0.974, a precision of 0.71, and a recall of 0.83 at the operational threshold, yielding an F1-score of 0.77 on the held-out evaluation set. Table 1 summarises the key system characteristics. Table 1: System profile of the fraud detection use case. Dimension Description Model type XGBoost ensemble (250 trees, max depth 6, learning rate 0.05) Input features 87 features: transaction attributes, geospatial signals, device fingerprints, velocity features (1 h / 24 h / 7 d windows) Volume \approx 4.2M transactions/day; \approx 0.08% fraud prevalence Latency budget \leq 200 ms end-to-end (feature engineering + scoring + explanation) Model performance AUC-ROC: 0.974; Precision: 0.71; Recall: 0.83; F1: 0.77 Deployment mode Substitution — automated blocking with optional analyst override Regulatory context EU PSD2 (SCA, dispute resolution obligations), GDPR Art. 22 (right to explanation), EBA fraud reporting guidelines 2.2 Stakeholder Analysis Three primary stakeholder groups interact with the system’s outputs and impose distinct, partially competing explainability requirements: • Compliance officers and regulators require tamper-evident, auditable logs of model decisions that are verifiable under GDPR Article 22 and defensible in regulatory reporting to the EBA. Their primary concerns are auditability and traceability : the ability to reconstruct, after the fact, the reasoning underlying any individual blocking decision. • Fraud analysts and customer service agents constitute the operational user base. Analysts reviewing flagged transactions require comprehensible, feature-level explanations that accelerate accurate override decisions. Customer service agents responding to cardholder disputes require concise, non-technical rationales suitable for external communication. Both sub-groups demand high comprehensibility and actionability . • Data scientists and ML engineers are responsible for model monitoring, retraining, and validation. They require technical transparency tools supporting the identification of feature importance drift, the debugging of systematic false-positive clusters (e.g., by merchant category or geolocation), and the validation of model behaviour under distributional shift. Their requirements prioritise fidelity and debuggability . This stakeholder structure maps directly and cleanly onto the three ESS axes: Compliance ( C ), User ( U ), and Developer ( D ). Notably, the dual user population—professional analysts and cardholder-facing agents—introduces heterogeneity within the User axis that is not fully captured by a single aggregated U score; this limitation is discussed in