Contrastive Analysis of Linguistic Representations in Large Language Model Outputs through Structured Synthetic Data Generation and Abstracted N-gram Associations
Sofía A. Desimone, Laura Alonso Alemany · 2026
Read Contrastive Analysis of Linguistic Representations in Large Language Model Outputs through Structured Synthetic Data Generation and Abstracted N-gram Associations by Sofía A. Desimone, Laura Alonso Alemany (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17398.txt.
Introduction
Desimone and Alonso offer a method for finding subtle bias in language-model output without a fixed list of suspect words. Controlled synthetic texts form minimal pairs that differ only in the social group named, keeping story conditions comparable. Word forms are then generalized, and a mutual-information variant flags forms that appear far more often for one group. A ranking step surfaces the most concentrated passages for expert review, linking stats to qualitative judgment.
Catalog record
| CREATOR | Sofía A. Desimone, Laura Alonso Alemany |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32640 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17398
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Contrastive Analysis of Linguistic Representations in Large Language Model Outputs through Structured Synthetic Data Generation and Abstracted N-gram Associations Sofía A. Desimone, Laura Alonso Alemany Abstract 1 Introduction and Motivation Large Language Models (LLMs) have demonstrated outstanding performance in text generation tasks. However, numerous studies have pointed out that these systems may reproduce and amplify biases present in their training data, particularly along dimensions such as gender, race, sexual orientation, and religion [ 6 , 14 , 21 , 8 , 6 , 14 , 21 , 11 , 8 ] . In a context where LLMs are rapidly being integrated into high-impact social domains, such as healthcare, education, and justice, the rigorous evaluation of their discursive behavior becomes a priority issue [ 4 , 24 , 18 ] . Much of the literature has focused on explicit forms of bias, in which marginalized groups are directly named and associated with identifiable stereotypes. However, bias may also operate implicitly, through more subtle and indirect mechanisms that shape model outputs [ 12 , 2 , 10 ] . These forms of bias often manifest through complex textual patterns rather than overtly negative statements, including differences in the attribution of roles, agency, or evaluative traits. For instance, certain groups may be disproportionately framed through narratives that are superficially positive yet ultimately limiting, such as overly inspirational or patronizing portrayals, while others are more frequently associated with attributes like professionalism, competence, or precision [ 9 , 23 ] . Such patterns may also emerge through differential associations with activities, emotional registers, or social roles, including both overrepresentation and systematic absence These manifestations are inherently more difficult to detect, particularly when analysis relies on predefined lists of sensitive terms, toxicity metrics, or fixed categorical frameworks (e.g., emotion labels such as fear, joy, or positivity). While useful in specific settings, these approaches tend to fragment the phenomenon and impose a reductionist structure that may obscure the broader discursive configurations through which subtle biases are expressed [ 15 , 18 ] . In this work, we propose a methodology aimed at identifying subtle, systematically differentiated patterns in generated text through contrastive synthetic data and statistical analysis of associations. Our approach seeks to identify textual fragments that concentrate linguistic configurations differentially associated with social groups under equivalent contextual conditions. These fragments are then assessed by experts to determine whether they may produce harms. The workflow consists of three main stages: (1) contrastive generation of texts for each social group; (2) statistical discovery of association patterns between social groups and textual expressions; and (3) identification of textual fragments with a high density of differentially associated expressions, so that experts may assess whether they produce any kind of harm. This design allows experts to focus on fragments that exhibit strong differential behaviors, facilitating the identification and characterization of discursive patterns associated with unequal or potentially harmful representations. In contrast to previous approaches, which often require inspecting large volumes of text or decontextualized outputs, our method directs attention to a reduced set of context-rich and statistically grounded examples. The rest of the paper is organized as follows. In the next Section, we describe the capabilities and limitations of different approaches to bias detection, and how this approach proposes to overcome them. Then, we describe the main parts of the proposed methodology: contrastive generation of texts, discovery of associations and fragment ranking and assessment. Finally, we present a proof of concept with the group of people with disabilities in Spanish. 2 Relevant Work Research on bias in language models has received increasing attention due to both the technical challenges it poses and its social implications [ 4 , 8 , 12 ] . Numerous studies have developed methods to explore, quantify, and mitigate bias in automated linguistic systems, consolidating an active field of evaluation and auditing. A substantial portion of this literature has focused on detecting explicit bias, frequently through the use of predefined lists of sensitive terms, stereotypical templates, or structured benchmarks [ 25 , 17 , 16 , 19 ] . While these approaches have proven effective in identifying overt forms of discrimination, they present limitations in capturing more nuanced phenomena, such as differential narrative framing, implicit condescension, or systematically biased representations that are not expressed through openly offensive language. In this regard, recent work emphasizes that some of the most persistent forms of bias in LLMs are implicit or structural [ 12 , 18 , 2 , 10 , 3 ] , operating through recurrent associations and discursive patterns that manifest indirectly. In response to these limitations, more contextualized approaches have begun to emerge, aimed at identifying patterns that arise beyond predefined categories. In this work, we propose a methodology aimed at operationalizing qualitative evaluation and collective participation, systematically producing evidence that can be critically examined by domain experts and potentially affected communities. In this way, the methodology structures the exploration process and the detection of statistically significant patterns, while experts are only required to assess the potential harms present in the identified fragments. In contrast with other approaches, our method does not predefine bias indicators; instead, these are discovered from the texts themselves and are not constrained to any fixed or predefined form. Rather, they may take arbitrary linguistic configurations. Moreover, expert involvement always occurs at a level of abstraction or concreteness that is natural to them and in a contextualized manner: by describing groups and situations that may be differentially represented by LLMs, and by evaluating the potential harm in fragments embedded within complete texts. 3 Contrastive Text Generation The first step in our methodology consists in generating synthetic data to obtain controlled contexts in which differential behavior across social groups may emerge. By constructing minimal pairs of prompts that vary only in the social group referenced, the methodology enables the systematic analysis of whether and how LLMs produce textual outputs for different groups under equivalent conditions. This approach is based on the assumption that subtle biases often emerge only through comparison: linguistic patterns that appear neutral in isolation, like the word ” inspiring ”, may reveal discriminatory effects when we can systematize how they are differentially distributed across texts for different social groups, and how they may occupy the space of other expressions and concepts, like ” reliable ”. The process is organized around two main components: (i) a set of scenarios that define the situational context and remain constant across conditions, usually originating from initial hypotheses from experts; and (ii) a set of markers that specify attributes of a person, distinguishing between the group of interest and the control groups. From a base prompt like the one shown in Figure 2 , pairs of texts are generated that share the same situational context but differ only in the marker used. Thus, we obtain minimal pairs of texts. The methodology of minimal pairs has been extensively used in bias measurements (e.g., Nangia et al. [ 17 ] ), but these minimal pairs are crucially contextualized, in contrast with isolated words or sentences. This contrastive design enables direct comparison between comparable narratives because the texts resulting from minimal pairs of prompts are comparable texts, facilitating the identification of patterns that occur disproportionately in texts associated with