Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations

Pardis Sadat Zahraei, Ali Emami · 2025

Read Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations by Pardis Sadat Zahraei, Ali Emami (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00748.txt.

Introduction

Translation tools can warp facts about gender when they move between tongues that mark gender and tongues that do not. Pardis Sadat Zahraei and Ali Emami build a test set for that warp in a 2025 paper. Their set holds 3,950 hard scenes across six languages, such as Persian and Finnish, that leave gender open.

Four major systems all slid into stereotypes on the set. Each one reached for male pronouns when a role hinted at status, some four to six times more than female ones. A mid-size open model, retrained on the new set, shed most of those errors. It went on to beat the big closed systems. The work shows targeted data can fix biased output.

Catalog record

CREATORPardis Sadat Zahraei, Ali Emami
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00748

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations Pardis Sadat Zahraei, Ali Emami Abstract Addressing gender bias and maintaining logical coherence in machine translation remains challenging, particularly when translating between natural gender languages, like English, and genderless languages, such as Persian, Indonesian, and Finnish. We introduce the Translate-with-Care (TWC) dataset, comprising 3,950 challenging scenarios across six low- to mid-resource languages, to assess translation systems’ performance. Our analysis of diverse technologies, including GPT-4, mBART-50, NLLB-200, and Google Translate, reveals a universal struggle in translating genderless content, resulting in gender stereotyping and reasoning errors. All models preferred masculine pronouns when gender stereotypes could influence choices. Google Translate and GPT-4 showed particularly strong bias, favoring male pronouns 4-6 times more than feminine ones in leadership and professional success contexts. Fine-tuning mBART-50 on TWC substantially resolved these biases and errors, led to strong generalization, and surpassed proprietary LLMs while remaining open-source. This work emphasizes the need for targeted approaches to gender and semantic coherence in machine translation, particularly for genderless languages, contributing to more equitable and accurate translation systems. 1 1 1 The dataset, code, and fine-tuned models are publicly available at: GitHub Repository , TWC Dataset , mBART-ft-TWC Model , and mBART-id-ft-TWC Model . Figure 1: Comparison of GPT-4, on TWC instances with the performance of mBART-ft-TWC, a fine-tuned version of the mBART-50 model on the TWC dataset. 1 Introduction Resolving semantic ambiguity, which refers to the presence of multiple possible interpretations of a word or phrase, is a central challenge in Machine Translation (MT). Recent benchmarks have revealed the limitations of conventional Neural Machine Translation (NMT) systems in handling ambiguous sentences, with a notable gap in performance on such cases ( Raganato et al. 2020 ; Campolungo et al. 2022 ) . Large Language Models (LLMs) offer a promising new direction, exhibiting strengths that have recently surpassed traditional NMT approaches ( Brown et al. 2020 ; Chowdhery et al. 2023 ; Huang et al. 2023 ) . These models have also demonstrated proficiency in resolving cases of semantic ambiguity, such as polysemous words and infrequent word senses, particularly in well-resourced language pairs such as English-Chinese and English-Russian ( Iyer et al. 2023 ) . Despite these advancements, various aspects of machine translation involving disambiguation remain challenging for LLMs, in areas such as pronominal coreference resolution ( Emelin and Sennrich 2021 ) or gender-neutral translation ( Savoldi et al. 2024 ) . Moreover, these issues become even more pronounced in low-resource languages, where the scarcity of diverse training data and the inherent complexity of these languages, such as complex morphology and syntax, pose significant challenges to current language model architectures, leading to poor generalization and performance degradation ( Agrawal et al. 2024 ; Khiu et al. 2024 ) . Our study explores a novel intersection of these two challenges: Translating genderless languages, which cover a wide spectrum including many low- to mid-resource languages, namely Persian, Indonesian, Finnish, Turkish, Estonian, and Azerbaijani, into natural gender languages like English. This task presents unique problems, including mitigating gender bias, maintaining neutrality when needed, and ensuring logical coherence in the translated text. Unlike previous work that primarily focused on gendered languages (e.g., Spanish, French), our work specifically targets genderless languages and builds upon existing evaluation benchmarks like WinoMT ( Savoldi et al. 2021 ) and MT-GenEval ( Currey et al. 2022 ) , which have been instrumental in assessing MT systems’ performance on gender-related translation tasks. As illustrated in Figure 1 , even state-of-the-art LLMs such as GPT-4 ( OpenAI 2023 ) struggle with multiple challenges when translating from genderless to natural gender languages. Specifically, a) they face difficulties with reasoning when choosing a pronoun that logically aligns with contextual clues, b) they struggle to maintain neutrality when the source language does not provide sufficient context to resolve the ambiguity, and c) they often exhibit bias , defaulting to gender stereotypes when translating non-gendered pronouns into gendered ones. Such biased or illogical translations can perpetuate harmful stereotypes, undermine trust in these systems, and impede effective cross-cultural communication. Our work not only identifies these challenges but also proposes solutions that could significantly improve the equity and accuracy of translation systems, particularly for genderless languages. The key contributions include: • Translate-with-Care (TWC) Dataset: We present TWC, a novel collection of 3,950 challenging translation scenarios across six low- to mid-resource languages, designed to evaluate machine translation systems on handling content from genderless to natural gender languages while avoiding gender bias and preserving logical coherence. • Detailed Analysis of Translation Models: We analyze diverse open-source and proprietary translation technologies, revealing a universal struggle with genderless content. All models prefer masculine pronouns when gender stereotypes could influence choices, with Google Translate and GPT-4 showing particularly strong bias (favoring male pronouns 4-6 times more in leadership and professional contexts). We also identify a concerning trend of content omission, with models failing to translate up to 32% of text in sentences that require reasoning to disambiguate. • Effective Model Enhancement: We demonstrate that fine-tuning mBART-50 on TWC substantially resolves the biases and errors identified in our analysis, outperforming larger, more recent models, including proprietary LLMs (see Figure 1 for an example). Our fine-tuned model also shows strong generalization to out-of-distribution languages, offering a promising direction for more fair and accurate translation systems. Challenge Type Language Source Sentence - Gloss (interlinear) - Ground Truth Translation Reasoning Indonesian \clubsuit Isabella menyukai olahraga dan Ali menyukai musik ; dia ingin menjadi atlet profesional . Isabella like sport and Ali like music ; he/she want to become athlete professional . Isabella loves sports and Ali loves music ; she wants to be a professional athlete . Bias Finnish \spadesuit Kuningas on muslimi ja kuningatar kristitty . Hän on avulias ja ystävällinen . King is Muslim and queen Christian . He/She is helpful and kind . The king is a Muslim and the queen is a Christian . One is helpful and kind . Neutrality Turkish \blacklozenge Kızım ve oğlum çalışıyor , ve o yorgun My daughter and my son working , and he/she happy . My daughter and son are working , and one is happy . Table 1: Examples of TWC categories with their original language statements, glosses, and ground truth translations. For improved clarity, the text is color-coded as follows: blue for pronouns, green for nouns, red for verbs, and purple for adjectives. Symbols indicate antecedent types: \clubsuit for Personal Names, \spadesuit for Titles, and \blacklozenge for Roles. 2 The TWC Translation Task TWC is designed to address the challenges associated with translating genderless languages and mitigating the biases and inconsistencies that arise during pronoun resolution. It consists of a collection of sentences in genderless languages, paired with their English translations, annotated to highlight the specific translation challenges they represent. Each TWC instance is represented as Q=\{L,M,A_{1},A_{2},P,R,T,C\} , where: • L : Source genderless language • M : The source sentence • A_{1},A_{2} : Candidate antecedents • P : Target pronoun • R : The correct English-translated pronoun antecedent among the choices A_{1} and A_{2} • T : The ground-truth translation (English), using R as the designated pronoun • C : Translation challenge type among the challenges of {Bias, Neutrality,

Read the complete text