Are All Spanish Doctors Male? Evaluating Gender Bias in German Machine Translation
Michelle Kappl · 2025
Read Are All Spanish Doctors Male? Evaluating Gender Bias in German Machine Translation by Michelle Kappl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19104.txt.
Introduction
Michelle Kappl, writing alone in 2025, builds a test for gender bias in German translation. German marks gender in grammar, which makes job titles a sharp test case. Her test set holds two hundred eighty-eight balanced sentences, labeled with real labor data on which jobs skew male or female.
Five common translation systems and one large language model were scored on the set. Most systems showed lasting bias toward male forms for many jobs. The language model did best, beating the classic systems. The dataset and scoring code are public, so others can repeat or extend the test.
Catalog record
| CREATOR | Michelle Kappl |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-19104
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Are All Spanish Doctors Male? Evaluating Gender Bias in German Machine Translation Michelle Kappl Abstract We present WinoMTDE, a new gender bias evaluation test set designed to assess occupational stereotyping and underrepresentation in German machine translation (MT) systems. Building on the automatic evaluation method introduced by Stanovsky et al. 2019 , we extend the approach to German, a language with grammatical gender. The WinoMTDE dataset comprises 288 German sentences that are balanced in regard to gender, as well as stereotype, which was annotated using German labor statistics. We conduct a large-scale evaluation of five widely used MT systems and a large language model. Our results reveal persistent bias in most models, with the LLM outperforming traditional systems. The dataset and evaluation code are publicly available at https://github.com/michellekappl/mt_gender_german . 1 Introduction In a globalized world, millions rely on machine translation (MT) systems to break language barriers in medicine, business, and diplomacy every day ( Vieira et al. 2021 ) . However, when these systems fail, the consequences can be severe ( Canfora and Ottmann 2020 ) . The results of a study conducted by Patil and Davies 2014 show Google Translate incorrectly translating the phrase “Your child is fitting” (which denotes a child having seizures) into the Swahili equivalent of “Your child is dead” . While translation errors in medical contexts can lead to life-threatening misunderstandings, it is not the only domain where MT systems can fail. Another example is depicted in Figure 1 , where Google Translate mistranslates a sentence from German to Spanish. Die Managerin f feuerte den Reiniger m , weil sie wütend war. 1 1 1 Translation: The manager f fired the cleaner m , because she was mad. El gerente m despidió a la limpiadora f porque estaba enojada. Figure 1: Example of gender bias in German Machine Translation by Google Translate, where occupational stereotypes are reinforced. In this case, the German noun Die Managerin , explicitly marked as female, was mistranslated into the masculine Spanish term El gerente . Despite clear grammatical markers indicating the subject’s gender, the MT system defaulted to a male translation, thereby producing a flawed translation. These phenomena are referred to as gender bias in MT and of rising concern in the field of natural language processing ( Savoldi et al. 2021 ; Costa-jussà 2019 ; Blodgett et al. 2020 ) . Bias Statement ( Blodgett et al. 2020 ) . Gender-biased translations reinforce societal assumptions about the roles and abilities of different genders ( Vervecken and Hannover 2015 ; Sterling et al. 2020 ) . If MT models systematically misrepresent female subjects and reinforce stereotypical gender roles in occupational contexts, they contribute to the invisibility of women in professions traditionally dominated by men ( Horvath et al. 2016 ) . Research has shown that children are particularly susceptible to such biases, which can shape their perceptions of career difficulty, prestige, and self-efficacy ( Vervecken and Hannover 2015 ) . Furthermore, Vervecken and Hannover 2015 found that using pair-forms (e.g., “Feuerwehrmänner und Feuerwehrfrauen” for male and female firefighters) instead of male generics increases children’s confidence in pursuing non-traditional careers. Studies also highlight a correlation between women’s self-efficacy in STEM (Science, Technology, Engineering, and Mathematics) occupations and the persistent gender pay gap ( Sterling et al. 2020 ) . Contribution. To address these issues and minimize potential harm, it is crucial to deepen our understanding of gender bias in MT. Prior research has primarily focused on English MT models, with Stanovsky et al. 2019 conducting the first large-scale evaluation on this topic. This work aims to bridge existing research gaps by introducing a German gender bias evaluation testset (GBET), WinoMTDE, which extends the proposed automatic evaluation method developed by Stanovsky et al. 2019 to German. The dataset is designed to evaluate occupational stereotyping and gender bias in German MT and therefore enabled us to do a systematic analysis of five widely used MT systems namely Google Translate, Microsoft Translator, Amazon Translate, DeepL, and SYSTRAN. In addition to these traditional MT systems, we also assess GPT-4o-mini, as large language models are increasingly integrated into everyday applications and frequently used for translation tasks ( Chan and Tang 2024 ) . These models were evaluated on their ability to correctly translate sentences from German to seven target languages that heavily exhibit gender in their grammatical structure: French, Italian, Spanish, Ukrainian, Russian, Arabic, and Hebrew. Unlike English, German uses explicit grammatical gender markers, which should, in theory, reduce ambiguity when translating into other gendered languages. One might expect MT systems to produce more accurate and gender-consistent translations due to these grammatical cues. However, despite the availability of such markers, our findings reveal that gender bias persists in most models. This indicates that the problem stems from systemic biases within model architectures and training data rather than source-language ambiguity. Related Work. Stanovsky et al. 2019 conducted the first large-scale evaluation of gender bias in English MT systems. They introduced the WinoMT GBET, which is based on two corpora of sentences following the Winograd schema ( Levesque et al. 2012 ) , namely Winogender ( Rudinger et al. 2018 ) and WinoBias ( Zhao et al. 2018 ) . In their evaluation, they found that all tested MT systems exhibited significant stereotypical and gender bias. 2 Methodology In this section we introduce the WinoMTDE dataset, discuss the evaluation pipeline, and outline the used metrics. 2.1 WinoMTDE We introduce the WinoMTDE 2 2 2 available at https://github.com/michellekappl/mt_gender_german dataset, a German GBET which is a translated subset of WinoMT by Stanovsky et al. 2019 . The dataset consists of 288 German sentences structured according to the Winograd schema (see Figure 1 ), where each sentence consists of a clearly gendered subject of interest (e.g. Die Managerin ) in the main clause, as well as another subject of opposite gender (e.g. der Reiniger ). A pronoun (e.g., sie ) in a dependent clause refers to the subject of interest. WinoMTDE currently includes only binary-gendered terms and pronouns. It does not account for non-binary pronouns or neutral occupational terms. Each sentence is annotated with: • The subject’s gender (male or female). • Its position in the sentence. • The stereotype alignment , i.e. if the occupation is pro- or anti-stereotypical. The dataset is balanced in regard to gender, containing an equal number of female and male-gendered subjects of interest (144 each). Stanovsky et al. 2019 used statistics from the U.S. Department of Labor to split WinoMT into equal parts pro- and anti-stereotypical instances. This is used for further evaluating each MT model regarding stereotypical gender bias. For the WinoMTDE testset to better reflect the German society, statistics from the German Department of Labor (Bundesagentur für Arbeit) were used. Each occupation of the WinoMTDE set was classified according to the “German Classifications of Occupations 2010 - Revised Version 2020” ( Statistik der Bundesagentur für Arbeit 2020 ) . This classification can be found in the appendix (see A.1 ). By considering the gender distribution of each classified occupation, the stereotypical gender (defined as more than 50%) associated with each occupation was determined. For example, the female occupation Managerin falls under the category “711 - Geschäftsführung und Vorstand” (managing and board members). Given that