Do Fair Models Reason Fairly? Counterfactual Explanation Consistency for Procedural Fairness in Credit Decisions

Gideon Popoola, John Sheppard · 2026

Read Do Fair Models Reason Fairly? Counterfactual Explanation Consistency for Procedural Fairness in Credit Decisions by Gideon Popoola, John Sheppard (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12701.txt.

Introduction

Popoola and Sheppard expose hidden procedural bias in outcome-fair credit models. Equal outcome scores do not guarantee that models reason the same way for different groups. Existing outcome-fair models can still reason differently about individuals. Their Counterfactual Explanation Consistency setup detects and mitigates that bias by aligning feature weights between individuals and their twins. They add a nearest-neighbour counterfactual making method, a new base for summed gradient checks, a single-person process fairness metric and a training loss.

Catalog record

CREATORGideon Popoola, John Sheppard
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63999
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12701

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Fair Models Reason Fairly? Counterfactual Explanation Consistency for Procedural Fairness in Credit Decisions Gideon Popoola, John Sheppard Abstract Machine learning algorithms in socially sensitive domains (e.g., credit decisions) often focus on equalizing predictive outcomes. However, satisfying these metrics does not guarantee that models use the same reasoning for different groups. We show that existing outcome-fair models can still apply fundamentally different reasoning to individuals, a “hidden procedural bias” missed by standard fairness metrics and algorithms. We propose Counterfactual Explanation Consistency (CEC), a framework that detects and mitigates this bias by aligning feature attributions between individuals and their counterfactual counterparts. Key contributions include a nearest-neighbor counterfactual generation method, a modified baseline for integrated gradient comparisons, an individual-level procedural fairness metric, and a corresponding training loss. We introduce a taxonomy identifying “Regime B” (same outcome, different reasoning) as a critical blind spot. Experiments on synthetic data, German Credit, Adult Income, and HMDA mortgage data demonstrate that outcome-fair baselines exhibit substantial hidden bias, while CEC substantially reduces it with modest utility cost. Introduction Machine learning (ML) models are now widely deployed in socially sensitive domains such as financial services for credit scoring, loan underwriting, and risk assessment ( 7 ; 13 ) . Regulatory frameworks such as the Equal Credit Opportunity Act (ECOA) and the Fair Housing Act prohibit lenders from disadvantaging individuals based on protected attributes, including race, gender, and ethnicity ( 1 ; 2 ) . Ensuring that ML models comply with these requirements has become a central challenge for practitioners and regulators alike. Most algorithmic fairness research addresses this challenge through outcome fairness , which focuses on comparing predictive outcomes across groups using criteria such as demographic parity or equalized odds ( 18 ; 15 ) . While effective at reducing outcome disparities, these metrics leave a critical question unanswered, which is does the model rely on the same reasoning when evaluating individuals from different groups? This question has deep roots in theories of procedural justice. In the philosophical and legal traditions, fairness requires not only equitable outcomes but also consistent application of decision criteria ( 30 ; 23 ; 35 ) . The principle that “like cases should be treated alike” demands that individuals with comparable qualifications be evaluated using the same standards, regardless of demographic membership. In lending regulation, ECOA explicitly prohibits disparate treatment even when outcomes appear equal ( 1 ) . A lender who evaluates Male applicants primarily on credit score but evaluates Female applicants primarily on employment history has engaged in prohibited conduct, regardless of whether approval rates are balanced. This legal and ethical framework motivates our focus on procedural fairness in automated decision systems. Figure 1 illustrates the core problem. Two loan applicants with nearly identical financial profiles but different protected groups both receive approval, satisfying outcome fairness. However, examining the model’s procedure reveals different reasoning. Applicant A’s decision is driven by credit score and income, while Applicant B’s decision depends primarily on employment history and collateral. This hidden procedural bias (different reasoning pathways producing the same outcome) is invisible to most traditional fairness metrics yet constitutes disparate treatment under ECOA. Applicant A (Group 0) Income=$60K, Score=720, Approved Credit Score 0.55 Income 0.28 Empl. History 0.10 Collateral 0.05 Other 0.02 Applicant B (Group 1) Income=$62K, Score=725, Approved Credit Score 0.08 Income 0.02 Empl. History 0.48 Collateral 0.38 Other 0.04 Outcome Fair ✓ Same decision Procedurally Unfair \times Different reasoning Figure 1: Two financially similar applicants from different demographic groups receive the same loan approval (outcome-fair), but the model’s procedure reveals entirely different reasoning (procedurally unfair). This Regime B bias is invisible to standard fairness metrics. We formalize this phenomenon through a taxonomy of four fairness regimes based on whether a model’s predictions and explanations remain consistent under counterfactual changes to the protected attribute (Table 1 ). For an individual (x,y,a) with counterfactual \tilde{x} from the opposite group, we assess two properties: prediction consistency (does the classification remain the same?) and explanation consistency (does the model’s reasoning remain the same?). The cross-product yields four regimes. Existing fairness metrics target Regimes C and D, where outcome disparities are observable. However, Regime B, in which predictions agree but reasoning differ is equally problematic from both legal and ethical standpoints, yet remains invisible to all standard metrics. Table 1: Taxonomy of fairness regimes. CEC targets Regime B, which is invisible to outcome-based methods. Regime Same Prediction? Same Reasoning? A (Fully Fair) ✓ ✓ B (Hidden Bias) ✓ \times C (Outcome Unfair) \times ✓ D (Severely Unfair) \times \times To address this gap, we propose Counterfactual Explanation Consistency (CEC), a framework for detecting and mitigating hidden procedural bias. The core idea is to compare integrated gradient attribution vectors between a factual individual and a counterfactual counterpart. If the model is procedurally fair, these attribution vectors should be similar. The model should weigh credit score, income, and other financial features in the same proportions regardless of which group the applicant belongs to. When they diverge, CEC quantifies the degree of hidden bias and provides a differentiable training signal to mitigate it. In this paper, we make the following contributions: • We formalize hidden procedural bias and introduce a 2×2 taxonomy distinguishing outcome vs. procedural fairness violations. • We propose a nearest-neighbor counterfactual generation method that produces realistic matches by controlling for financial merit and creditworthiness outcome, without requiring causal graphs or structural equations. • We propose Counterfactual Explanation Consistency (CEC) , a metric that measures explanation stability under demographic counterfactuals, along with a differentiable training loss that jointly optimizes accuracy, outcome fairness, and explanation consistency. • We conduct a comprehensive evaluation across synthetic, benchmark, and real-world lending datasets demonstrating that outcome-fair baselines contain substantial hidden bias, which CEC substantially reduces with minimal utility cost. Related Work Outcome Fairness in Machine Learning The algorithmic fairness literature has developed numerous statistical criteria for evaluating predictive equity. Demographic parity requires equal positive prediction rates across groups ( 15 ) ; equalized odds requires equal true positive and false positive rates ( 18 ) ; and calibration requires that predicted probabilities reflect true outcomes within each group ( 9 ) . These criteria are known to be mutually incompatible in general ( 21 ) , leading to a rich literature on navigating trade-offs. Mitigating bias spans the machine learning pipeline. Pre-processing methods transform training data to remove correlations with protected attributes ( 15 ; 20 ; 28 ) . In-processing methods incorporate fairness constraints during training through reductions ( 3 ) , adversarial objectives ( 39 ) , or constrained optimization ( 11 ) . Post-processing methods adjust decision thresholds after training ( 18 ) . While these approaches effectively reduce outcome disparities, they evaluate only what a model predicts, not how it arrives at that prediction. Our work shows that this gap allows models to harbor hidden procedural bias even when all outcome constraints are satisfied. Explainability and Fairness Post-hoc explanation methods such as Local Interpretable Model-Agnostic Explanations (LIME) ( 31 ) , SHapley Additive exPlanations (SHAP) ( 24 ) , and integrated gradients (IG) ( 34 ) have become essential tools for understanding model behavior in high-stakes domains. A growing body of work connects explainability with fairness. 12 ( 12 )

Read the complete text