UGID : Unified Graph Isomorphism for Debiasing Large Language Models

Zikang Ding, Junchi Yao, Junhao Li, Yi Zhang, Wenbo Jiang, Hongbo Liu, Lijie Hu · 2026

Read UGID : Unified Graph Isomorphism for Debiasing Large Language Models by Zikang Ding, Junchi Yao, Junhao Li, Yi Zhang, Wenbo Jiang, Hongbo Liu, Lijie Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19144.txt.

Introduction

Ding and co-authors tackle social bias inside large language models, not just in their outputs. Their method, UGID, treats the Transformer as a graph of attention routes and hidden states. It forces that graph to stay the same when only a sensitive trait changes in the input. Added limits on biased word scores help keep core meanings intact. Tests show less bias in and out of the training setting. The team reports that safety and general skill are kept.

Catalog record

CREATORZikang Ding, Junchi Yao, Junhao Li, Yi Zhang, Wenbo Jiang, Hongbo Liu, Lijie Hu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42802
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19144

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UGID : Unified Graph Isomorphism for Debiasing Large Language Models Zikang Ding, Junchi Yao, Junhao Li, Yi Zhang, Wenbo Jiang, Hongbo Liu, Lijie Hu Abstract Large language models (LLMs) exhibit pronounced social biases. Output-level or data-optimization–based debiasing methods cannot fully resolve these biases, and many prior works have shown that biases are embedded in internal representations. We propose U nified G raph I somorphism for D ebiasing large language models ( UGID ), an internal-representation–level debiasing framework for large language models that models the Transformer as a structured computational graph, where attention mechanisms define the routing edges of the graph and hidden states define the graph nodes. Specifically, debiasing is formulated as enforcing invariance of the graph structure across counterfactual inputs, with differences allowed only on sensitive attributes. UGID jointly constrains attention routing and hidden representations in bias-sensitive regions, effectively preventing bias migration across architectural components. To achieve effective behavioral alignment without degrading general capabilities, we introduce a log-space constraint on sensitive logits and a selective anchor-based objective to preserve definitional semantics. Extensive experiments on large language models demonstrate that UGID effectively reduces bias under both in-distribution and out-of-distribution settings, significantly reduces internal structural discrepancies, and preserves model safety and utility. 1 Introduction (a) GPT-2. Bias distribution shifts. (b) LLaMA-3-8B. Bias distribution shifts. Figure 1 : Visualization of bias distribution shifts. The vertical layout allows for clear layer-wise comparison between GPT-2 (top) and LLaMA-3 (bottom). Large Language Models (LLMs) have become foundational infrastructure for modern artificial intelligence systems, yet they often reflect and amplify social biases latent in the training data ( 38 ; 14 ; 10 ; 22 ) . These biases are not merely surface-level artifacts but are deeply encoded within internal representations and computational processes ( 4 ; 5 ) . Existing debiasing work can be divided into (i) model-external methods , which adjust data, inference, or output behavior but do not constrain internal bias representations ( 40 ; 17 ; 15 ; 16 ; 9 ; 4 ) ; and (ii) model-internal methods , which primarily focus on locating or understanding local components like attention heads or FFN vectors ( 20 ; 12 ; 24 ; 39 ) . However, these existing methods are still difficult to fundamentally weaken the biases present inside the model ( 8 ; 31 ) . Our investigation reveals the scale-dependence of bias within the internal mechanisms of the model. Through controlled experiments, we observe that when constraints are imposed only on the attention mechanism, bias metrics can be significantly reduced in models with small parameter scales like GPT-2, but this effect is significantly weakened or even fails when the model scale is extended to the billion-parameter level (Figure 1 ). This indicates that bias is not determined solely by attention; instead, feed-forward networks (FFNs), serving as primary storage units for semantic and factual knowledge, also encode and amplify bias ( 29 ; 20 ) . Thus, constraining attention routing alone is insufficient to fundamentally weaken the bias embedded within the model. To overcome these limitations, we propose UGID , the first unified distance-invariant debiasing framework that reformulates the problem as a computational graph isomorphism challenge. Leveraging the perspective of mechanistic interpretability, we model the internal bias migration as a dynamic computational graph \mathcal{G}=(\mathcal{V},\mathcal{E}) ( 2 ) , where token representations constitute the node set \mathcal{V} ( 12 ; 20 ) and attention mechanisms define the weighted dependency edge set \mathcal{E} ( 3 ) . Specifically, we develop a dual alignment framework to achieve graph isomorphism: for edges, we employ a combinatorial Laplacian operator with attention-sink masking to align the spectral characteristics of semantic routing; for nodes, we propose a selective representation alignment strategy to prevent FFNs from bias compensation. Additionally, a log-space behavioral guidance is introduced to stabilize the model’s distribution over non-sensitive semantics. Extensive experiments across multiple scales demonstrate that UGID achieves state-of-the-art (SOTA) performance in both debiasing effectiveness and structural stability. Mechanistic analyses further verify that our framework effectively decouples bias from internal reasoning topology. The contributions of our work are summarized as follows: • We propose UGID , the first framework to achieve debiasing via Unified Graph Isomorphism. By employing Laplacian Spectral Constraints to precisely align the routing topology (edges) and Selective Anchoring to stabilize semantic memory (nodes), UGID mechanistically severs the migration pathways of bias. • We provide mechanistic interpretability evidence for the debiasing process. Through Spectral Diagnostics, Logit Lens, and Activation Patching, we visualize and verify the elimination of bias at a topological level, offering a physical-level explanation for the internal bias governance in LLMs. • Extensive experiments on public datasets demonstrate that UGID achieves a better trade-off between debiasing and model performance preservation, consistently outperforming state-of-the-art methods across various model sizes and data scenarios. Figure 2 : Overview of the UGID framework. 2 Related Work Debiasing and Performance. Social bias in large language models (LLMs) is not only reflected in their outputs, but is also embedded in the models’ internal representations and computational processes. These biases are complex and diverse, ranging from explicit stereotypes captured by benchmarks such as BBQ ( 23 ) and CrowS-Pairs ( 21 ) , to more subtle affective disparities observed in datasets like BOLD ( 7 ) and RealToxicityPrompts ( 11 ) . As a result, balancing effective debiasing with preserving model performance remains a fundamental challenge. Existing debiasing methods often face a bottleneck: stronger debiasing capability can come at the cost of the model’s original utility ( 5 ) . For example, data-optimization-driven approaches such as CDA ( 40 ) may reduce bias while degrading the model’s intrinsic performance ( 15 ) . In contrast, external methods such as Self-Debias ( 9 ) , KLAAD ( 16 ) , and BiasFilter ( 6 ) essentially intervene at the behavioral level. These approaches also struggle to maintain general reasoning ability and are vulnerable to prompt rephrasing or out-of-distribution (OOD) scenarios, because they do not fundamentally weaken the internal bias structures of the model. Mechanistic Analysis and Structural Interventions. Mechanistic interpretability offers a clearer lens to investigate these internal pathways, identifying bias-carrying attention heads via causal mediation ( 30 ; 3 ) and biased ”Key-Value Memories” in Feed-Forward Networks (FFNs) ( 12 ; 20 ) . Crucially, our research reveals that bias is scale-dependent: interventions on attention alone may effectively reduce bias in small-scale models like GPT-2 but fail in billion-parameter models like LLaMA-3-8B due to bias migration into FFN components. This phenomenon suggests that a robust debiasing framework must jointly address attention routing and hidden representations to achieve structural invariance. While recent research has explored graph-structured removal ( 1 ) , specific bias subspaces ( 35 ) , and causal adjustments ( 33 ; 36 ) , these methods typically focus on isolated components and fail to arrest the holistic migration of bias across architectural layers. Furthermore, although 18 advocates for limited interventions to preserve utility, and other studies leverage structured knowledge or internal representations ( 19 ; 37 ; 32 ) , they do not enforce the topological isomorphism necessary to handle out-of-distribution (OOD) scenarios or complex bias migration pathways. UGID addresses these gaps by reformulating debiasing as a unified structural alignment problem, ensuring that the reasoning topology

Read the complete text