SoS : Analysis of Surface-over-Semantics in Multilingual Text-To-Image Generation

Carolin Holtermann, Florian Schneider, Anne Lauscher · 2026

Read SoS : Analysis of Surface-over-Semantics in Multilingual Text-To-Image Generation by Carolin Holtermann, Florian Schneider, Anne Lauscher (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16803.txt.

Introduction

Carolin Holtermann, Florian Schneider, and Anne Lauscher test image bots across tongues. Such bots turn words into shots. Most users type in English. The same ask in a new tongue should give the same shot. Yet bots oft swap sense for surface cues. They paint stock scenes tied to the tongue's home land.

Their 2026 preprint names this flaw surface-over-semantics. It gauges how oft bots pick stock traits over true sense. The flaw can spread trite views of lands and folks. Clear gauges push bot makers to serve all tongues fair. The work aids fair AI for a world of many tongues.

Catalog record

CREATORCarolin Holtermann, Florian Schneider, Anne Lauscher
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16803

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SoS : Analysis of Surface-over-Semantics in Multilingual Text-To-Image Generation Carolin Holtermann, Florian Schneider, Anne Lauscher Abstract Text-to-image (T2I) models are increasingly employed by users worldwide. However, prior research has pointed to the high sensitivity of T2I towards particular input languages – when faced with languages other than English (i.e., different surface forms of the same prompt), T2I models often produce culturally stereotypical depictions, prioritizing the surface over the prompt’s semantics. Yet a comprehensive analysis of this behavior, which we dub Surface-over-Semantics (SoS) , is missing. We present the first analysis of T2I models’ SoS tendencies. To this end, we create a set of prompts covering 171 cultural identities, translated into 14 languages, and use it to prompt seven T2I models. To quantify SoS tendencies across models, languages, and cultures, we introduce a novel measure, and analyze how the tendencies we identify manifest visually. We show that all but one model exhibit strong surface-level tendency in at least two languages, with this effect intensifying across the layers of T2I text encoders. Moreover, these surface tendencies frequently correlate with stereotypical visual depictions. **Warning: This paper contains discussions about stereotypes** 1 Introduction With the public release of text-to-image (T2I) tools like Dall-E 23 , automatically generating images via prompting, e.g., for marketing purposes, has become a widely adopted practice for many users worldwide. Given that only about 20% of the global population speaks English fluently 1 1 1 https://www.ethnologue.com/insights/most-spoken-language/ , many of these users may prompt the systems in languages other than English. Accounting for this variety, while most T2I models are still primarily trained and tested on English only ( 27 , e.g., Stable Diffusion v2.1;) , several multilingual models have been presented ( 34 , e.g., Kandinsky-3;) , enabling wider linguistic inclusion. While intuitively, semantically equivalent prompts should produce similar outputs, previous work demonstrated that T2I models are sensitive to the particular input language: given the same prompt and its translation, the output may be highly different – often reflecting cultural stereotypes associated with the input languages ( 33 , e.g.,) . As such, 30 found that exchanging a single prompt character with a homoglyph from a different script may activate depictions of cultural elements tied to that character. Importantly, these studies demonstrate the existence of a tension between the surface form of an input (i.e., its language, its script, etc.), and the semantics of the prompt (i.e., the actual description of what should be visualized) . However, so far, all studies operate on a handful of languages and/or cultures only, and, surprisingly, they fail to quantify the tendencies between prompt surface and semantics. This gap directly hinders further research on controlling how much the model is guided by either of the two, and the development of globally inclusive and culturally fair T2I models. Figure 1: Overview of our evaluation setup: (1) Constructing and translating each prompt into 13 other languages; (2) Generating corresponding images using one of 7 T2I models; (3) Analyzing output images through the SoS Score, a color analysis, and an analysis of commonly occurring descriptive terms. We conduct the first systematic analysis of a model’s alignment with the surface form of a prompt versus its semantic meaning. To do this, we create a dataset covering 171 cultures and 14 languages, with translations provided by native speakers. Using this dataset, we generate images with seven T2I models and analyze their reliance on the input language vs. a culture mentioned. To quantify the models’ Surface-over-Semantics (SoS) tendencies, we introduce a novel evaluation measure – SoS score – which allows us to compare models and languages. We then examine how bias toward a prompt’s surface form manifests in visual outputs. Our study framework (Figure 1 ) enables us to answer the following research questions (RQs): (RQ1) Can we quantify the tension between surface and semantics? Yes. Lacking a suitable method to analyze SoS tendencies of models, we propose the SoS score (§ 4 ), an embedding-based method that measures the similarity of T2I’s generations towards average representations of a surface form and a semantic content of the input prompts. We show that it effectively captures SoS tensions in line with human perception. (RQ2) Do SoS tendencies differ across different models and languages? Yes. Applying our measure to quantify SoS tendencies (§ 5 ), we find that all but one of the tested models exhibit strong surface tendencies in at least two, and up to six, languages. These biases are amplified when generations are conditioned on representations from the upper layers of the text encoders in T2I models. (RQ3) Are observed SoS tendencies manifested in the concrete visual depictions generated? Yes. We couple our measurements with complementary analyses based on Visual Question Answering (VQA) (§ 6 ). We find that languages with strong surface tendencies according to their SoS scores often trigger culturally stereotypical depictions. 2 Related Work (Cultural) Bias in T2I Models Numerous studies have explored bias in T2I models; for a comprehensive overview, see 35 . Most prior research focused on sociodemographic biases in T2I, examining dimensions such as gender, race, age, and nationality 16 ; 1 ; 32 . Mitigation techniques include fine-tuning on diversified datasets through counterfactual data augmentation 2 , and training on newly generated diverse instances 5 . Cultural biases have only recently gained attention. 11 studied visual cultural stereotypes in depictions of individuals from different cultural backgrounds. 12 introduced a benchmark to assess cultural awareness and diversity, identifying significant performance gaps for certain cultures. Similarly, 38 presented a dataset of cultural concepts, including reference images to evaluate and fine-tune T2I models. However, these studies remain monolingual, overlooking the effect of the prompting language on image generation. Language-Induced Bias in AI Models Another line of research explores how prompting language affects the output of generative models, particularly in LLMs. 26 proposed a benchmark to evaluate the factual and combinatorial knowledge of LLMs across 44 languages. Other studies explored shifts in models’ cultural values due to multilingual model fine-tuning 4 , variations in moral norms across languages 8 , prompt language-specific stereotypes 18 , and LLM safety across multiple languages 36 . Additionally, 17 found a Western cultural bias when prompting LLMs in Arabic, i.e., models favoring the generation of entities associated with Western traditions and values. In contrast, work on language-induced bias in T2I remains limited. 7 showed how multilingual prompting can amplify gender stereotypes in T2I. 30 found that minor character substitutions in textual prompts with non-Latin homoglyphs can induce strong cultural biases. However, their analysis is restricted to a limited set of homoglyphs, thereby covering only a narrow subset of cultural variations. 33 prompted T2I models with cultural concepts while translating parts of the prompt into ten languages. Their analysis relies on CLIP-based measures and VQA to assess cultural representation and national associations in images. We evaluate a broader and more diverse range of cultures and languages, and propose a novel evaluation measure that operates independently of textual descriptions. 3 Dataset and Overall Setup We create a new multilingual prompt dataset in three steps: we (1) select 171 diverse cultures and 13 diverse languages (plus English); (2) define English prompt templates which we instantiate with explicit mentions of cultures; (3) let native speakers

Read the complete text