When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity
Nisrine Rair, Alban Goupil, Valeriu Vrabie, Emmanuel Chochoy · 2025
Read When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity by Nisrine Rair, Alban Goupil, Valeriu Vrabie, Emmanuel Chochoy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17548.txt.
Introduction
When people disagree about whether a text is offensive, a single accuracy score hides what the model has actually learned. Nisrine Rair, Alban Goupil, Valeriu Vrabie, and Emmanuel Chochoy turn to Mapper, a tool from topological data analysis that maps the shape of a model inner representation space. They apply it to a large RoBERTa model fine tuned on an offence data set where human annotators often disagreed.
Fine tuning, they find, reshapes the space into separate, curved regions that match the model own predictions, even on the most ambiguous texts. Nearly all regions were highly pure in prediction, yet agreement with the human labels dropped exactly where annotators had disagreed. Rair and her co-authors argue that Mapper exposes overconfident clusters and collapsing boundaries that standard plots miss.
Catalog record
| CREATOR | Nisrine Rair, Alban Goupil, Valeriu Vrabie, Emmanuel Chochoy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17548
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity Nisrine Rair, Alban Goupil, Valeriu Vrabie, Emmanuel Chochoy Abstract Language models are often evaluated with scalar metrics like accuracy, but such measures fail to capture how models internally represent ambiguity, especially when human annotators disagree. We propose a topological perspective to analyze how fine-tuned models encode ambiguity and more generally instances. Applied to RoBERTa-Large on the MD-Offense dataset, Mapper, a tool from topological data analysis, reveals that fine-tuning restructures embedding space into modular, non-convex regions aligned with model predictions, even for highly ambiguous cases. Over 98 % 98\% of connected components exhibit ≥ 90 % \geq 90\% prediction purity, yet alignment with ground-truth labels drops in ambiguous data, surfacing a hidden tension between structural confidence and label uncertainty. Unlike traditional tools such as PCA or UMAP, Mapper captures this geometry directly uncovering decision regions, boundary collapses, and overconfident clusters. Our findings position Mapper as a powerful diagnostic tool for understanding how models resolve ambiguity. Beyond visualization, it also enables topological metrics that may inform proactive modeling strategies in subjective NLP tasks. For reproducibility, all code and experiment configurations are released 1 1 1 https://github.com/NisrineRair/tda-nlp-representations . 1 Introduction Ambiguity is a persistent challenge in Natural Language Processing (NLP) ( Pavlick and Kwiatkowski 2019 ) when instances often permit multiple plausible meanings or evoke differing judgements across annotators ( Wan et al. 2023 ) . This is the case in offensive language detection or natural language inference, where annotators may disagree ( Leonardelli et al. 2021 ; Nie et al. 2020 ) . While prior work has focused on modeling the ambiguity using soft labels or confidence estimates ( Swayamdipta et al. 2020 ; Fornaciari et al. 2021 ) , less is known about how ambiguous examples are internally structured in representation space. (a) UMAP projection (b) Mapper graph Figure 1: Illustration of geometric (UMAP) and topological (Mapper) views of model representations. Figure( 1(a) ) UMAP shows apparent separation between offensive and non-offensive instances. ( 1(b) ) Mapper reveals a more fragmented and connected structure, with offensive and non-offensive regions interwoven, highlighting spatial organization that UMAP flattens. Fine-tuning of language models is known to rearrange data within the embedding spaces, increasing cohesion and alignment with task-relevant categories ( Zhou and Srikumar 2021 ; Rajaee and Pilehvar 2021 ) . These changes are often analyzed using geometric tools such as PCA and UMAP, and quantified using metrics like cosine similarity, silhouette score, and anisotropy ( Ethayarajh 2019 ; Coenen et al. 2019 ; Cai et al. 2021 ) . Yet such tools can oversimplify high-dimensional structure. PCA’s global projections compress variance into a limited number of axes ( Wold et al. 1987 ) , while UMAP enforces local Euclidean neighborhoods ( McInnes et al. 2020 ) , often at the expense of capturing nonlinear or hierarchical relationships. In contrast, linguistic embedding space has been shown to exhibit natural hyperbolic geometry Nickel and Kiela 2017 , a structure that topological methods can preserve without relying on Euclidean assumptions. This is particularly important in high-dimensional settings, as dimensionality increases, points tend to become nearly equidistant, blurring the notion of proximity and undermining the interpretability of distance-based measures ( Donoho 2000 ; Aggarwal et al. 2001 ) . As a result, familiar geometric intuitions about neighborhoods and separation begin to break down, making it easy to misread the structure of the embedding space. This limitation is illustrated in Figure 1(a) where UMAP applied to the final-layer [CLS] embeddings of a RoBERTa-Large model fine-tuned on binary offensive language detection, presents the offensive class as a single, compact cluster. On the contrary, topological tools construct graphs based on point connectivity rather than distances, revealing a more fragmented and nuanced organization as in Figure 1(b) , where offensive instances are spread across multiple distinct yet connected regions, suggesting that geometric projections may flatten meaningful topological structure. This example highlights a key limitation of classical tools: visual simplicity can come at the cost of structural fidelity. This limitation reflects the embedding space’s non-Euclidean, hyperbolic structure. In this work, we use Mapper, a topological tool, to analyze how fine-tuned language models embed ambiguous instances and how this structure manifests in the geometry of embedding space. Focusing on offensive language detection, we explore how fine-tuning reshapes topological organization, why models exhibit overconfidence in ambiguous inputs, and how these behaviors relate to human disagreement. Our results show that fine-tuning imposes topological regularity: even instances with annotation disagreements are mapped into smooth, consistent regions. Rather than ignoring uncertainty, language models appear to spatially reorganize it, resolving ambiguity into reliable structures. 2 Related Work Ambiguity, Disagreement, and Uncertainty in NLP. Ambiguity in NLP has been studied from multiple angles. Early work often treated disagreement as label noise to be filtered ( Zhang et al. 2017 ) , whereas more recent studies argue that it reflects genuine linguistic uncertainty and advocate for distributional supervision to capture annotator variability ( Pavlick and Kwiatkowski 2019 ; Mostafazadeh Davani et al. 2022 ) . Building on this perspective, several studies investigated model behavior on ambiguous inputs. ( Swayamdipta et al. 2020 ) show that ambiguous examples exhibit unstable training dynamics, while Liu et al. 2023 argue that models tend to be overconfident on such inputs, failing to reflect underlying uncertainty in their predictions. Other research has challenged the use of majority vote in subjective tasks. Leonardelli et al. 2023 argue that majority aggregation suppresses meaningful disagreement signals, motivating soft-label and annotator aware modeling. These approaches assess ambiguity at the output level, leaving open the question of how it is structurally encoded within the model. Fine-Tuning and Embedding Space. Fine-tuning is known to reshape representation in the embedding space. Studies using distance-based metrics show that fine-tuned models for classification tasks increase intra-class cohesion and expand inter-class separation ( Zhou and Srikumar 2021 ) . Spectral analyses reveal that variance becomes concentrated in relevant directions, reducing isotropy of embedded instances ( Cai et al. 2021 ; Ethayarajh 2019 ) . Probing classifiers further confirm that linguistic features become more linearly separable after fine-tuning ( Hewitt and Manning 2019 ) . However, distance-based metrics oversimplify or distort high dimensional structure, potentially missing connectivity patterns and latent organization. Topological Methods in NLP. Topological Data Analysis (TDA) provides powerful tools for studying high-dimensional data beyond geometry, focusing on how points connect rather than how far apart they are. Two key tools are (1) Mapper ( Singh et al. 2007 ) , which constructs graph networks by projecting data through a lens into lower dimensions, and (2) Persistent homology ( Carlsson 2009 ) , which quantifies topological features like clusters or holes that persist across scales. While both methods have demonstrated significant success in different applications, for example in identifying novel cancer subtypes from gene expression data ( Lum et al. 2013 ) , their application to NLP remains limited, with few applications that can be categorized into three main directions: (1) Lexical-level analysis , Rathore et al. 2023 employed Mapper to track topological transformations in word embedding spaces during fine-tuning, revealing how semantic relationships reorganize at different training stages. (2) Summarizing documents , ( Guan et al. 2016