Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Models
Veronika Smilga · 2025
Read Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Models by Veronika Smilga (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-06638.txt.
Introduction
Veronika Smilga tests bias leakage in small language models in a solo 2025 study. Semantic leakage means a model's answers get colored by associations nobody asked for. Prior work found it in large models above seven billion parameters. Smilga asks whether smaller models leak less simply from limited capacity.
Her test bed is the Qwen model family from half a billion to seven billion parameters. A new prompt set focuses on color associations, sorted by type. Smaller models did leak less overall than large ones. The trend was not smooth, though. Mid sized models sometimes out leaked their bigger siblings. Data, generations, and scoring code are all public. The study tempers hopes that shrinking alone cures associative bias.
Catalog record
| CREATOR | Veronika Smilga |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-06638
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Models Veronika Smilga Abstract Semantic leakage is a phenomenon recently introduced by Gonen et al. 2024 . It refers to a situation in which associations learnt from the training data emerge in language model generations in an unexpected and sometimes undesired way. Prior work has focused on leakage in large language models (7B+ parameters). In this study, I use Qwen2.5 model family to explore whether smaller models, ranging from 500M to 7B parameters, demonstrate less semantic leakage due to their limited capacity for capturing complex associations. Building on the previous dataset from Gonen et al. 2024 , I introduce a new dataset of color-focused prompts, categorized into specific types of semantic associations, to systematically evaluate the models’ performance. Results indicate that smaller models exhibit less semantic leakage overall, although this trend is not strictly linear, with medium-sized models sometimes surpassing larger ones in leaking behavior. The dataset, the model generations, and the evaluation code are publicly available at https://github.com/smilni/semantic_leakage_project . 1 Introduction Language Models (LMs) have become omni-present in our everyday lives. However, the way they function is still largely a black-box. Inherently influenced by the training data, LMs demonstrate human-like perception of right and wrong ( Schramowski et al. 2021 ) , while also exhibiting a variety of undesirable social biases ( Bolukbasi et al. 2016 ; Lucy and Bamman 2021 ) . Navigli et al. 2023 argue that the source of such biases are associations learnt by the model in the course of its training. A broader example of learnt associations surfacing in an unexpected and unsuitable way is semantic leakage ( Gonen et al. 2024 ) . The association learnt by the model might make a concept from a prompt influence the model generation out of proportion. That results in bizarre outputs, such as “The dinner was served on pink plates. Today’s dish was… rose petal soup ’’ 1 1 1 Generated by Qwen2.5-3B-Instruct . Another interesting case of semantic leakage is the leakage of literal word-by-word meaning of idioms and proper names, such as “Ivory is a student. Her favorite color is… white ’’ 2 2 2 Generated by Qwen2.5-1.5B-Instruct . Semantic leakage in LMs is closely related to problems in word-to-concept mapping in image generation. With prompts containing several objects, the attributes of the objects may leak into each other, get interchanged, or one of the objects may be omitted from the image altogether ( Feng et al. 2023 ; Dahary et al. 2024 ) . Conversely, the presence of one ambiguous word in a prompt may lead to the model producing a picture of several entities corresponding to each of the word’s meanings ( Rassin et al. 2022 ) . While the above-described problems with word-to-concept mapping problems lead to errors in image generation, one may argue that semantic leakage in text generation, on the opposite, helps in maintaining the coherence of the text. Both semantic leakage and bias in machine learning models seem to be rooted in undesirable associations learnt during training. Unlike inherent model bias, semantic leakage does not necessarily result in incorrect or harmful generations. However, as pointed out by Gonen et al. 2024 , semantic leakage is a common pattern that characterises language models across different sizes and architectures, large and small, instruction-tuned and general-purpose ones. It would thus be interesting to investigate the phenomenon further to understand how models internalize and use associations. Gonen et al. 2024 analyse semantic leakage in various categories with strong semantic association component, including colors, food, animals, songs, and occupations. They test several Large LMs, including models of Llama-2 7B+ and LLama-3 8B+ families as well as OpenAI’s GPT3.5, GPT4 and GPT4o. In this project, I further investigate the phenomenon of semantic leakage. Building on Gonen et al. 2024 who examined models of 7 billion parameters and larger, I choose to focus on smaller LMs, from 500 million to 7 billion parameters. The goal is to determine whether the model’s size affects its proneness to semantic leakage. I hypothesize that smaller models, due to their size, may be unable to capture the semantic associations that result in semantic leakage. Thus, I am expecting smaller LMs to exhibit less leaking behavior as compared to larger ones. Additionally, I build a new set of prompts focusing on the category of color and test the models both on the original dataset by Gonen et al. 2024 and the color-focused one. Color-related prompts are divided into three categories: 1) prompts with a mention of a color, with a non-color-related concept to be generated; 2) prompts with a mention of a color, with another color to be generated; 3) prompts with names or set expressions mentioning a color, with another color to be generated 3 3 3 Refer to Subsection 2.3 for example prompts of each category. . The goal is to test whether the models exhibit leakage more for any of the categories. I hypothesize that prompts containing a color with another color to be generated, i.e. prompts of type 2) and type 3), leak more as models may take short-cuts by repeating previously mentioned colors. 2 Methods 2.1 Models Qwen2.5 ( Team 2024 ) is a series of large language models sized from 500 million to 72 billion parameters, both general-purpose and instruction-fine-tuned. Qwen2.5 is the most capable state-of-the-art series of models with open weights and smaller counterparts available as of today 4 4 4 As of October 1st, 2024, according to Chatbot Arena LLM Leaderboard ( Chiang et al. 2024 ) , among models with open weights, Qwen2.5-72b-Instruct is only surpassed by Meta-Llama-3.1-405b-Instruct. However, Llama-3.1 model family only features models of 8B, 70B, and 405B parameters, while I aim to evaluate models of 7B and smaller. . I test all Qwen2.5-Instruct models up to 7B parameters: Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct-GPTQ-Int4 5 5 5 Dut to computational constraints, I had to use GPTQ-quantised ( Frantar et al. 2023 ) version of Qwen2.5-7B-Instruct that would fit into a single P100 GPU. . Only Instruct models are tested as, according to Gonen et al. 2024 , instruction-finetuned models exhibit more leaking behavior than their general-purpose counterparts. In addition to that, I found it hard to get meaningful generations from smaller non-instruction-finetuned Qwen models. All models were accessed using Huggingface 6 6 6 https://huggingface.com . Inference was run using Kaggle notebooks 7 7 7 https://www.kaggle.com on a single P100 GPU. Temperature was set to an intermediate value of 0.5 to ensure balance between exploring diverse token probabilities and preserving high-probability sequences. Maximum number of tokens to be generated was set to 10. Additionally, generated text was capped after the first full stop, exclamation or question mark, following the original paper. 2.2 Evaluation metrics Cat. Prompt type Generation type Example prompt & generation 1 Color Non-color-related item He likes red . He works as a… firefighter 2 Color Color White clouds floated in the summer sky. The fence was painted… white and weathered 3 Color-related name or set expression Color Mary saw the world through rose-colored glasses . She was wearing a T-shirt colored… pink Table 1: Dataset categories. Color-related concepts in prompt are in black bold. Color-related concepts in model