Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translation

Dayeon Ki Kevin Duh Marine Carpuat · 2025

Read Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translation by Dayeon Ki Kevin Duh Marine Carpuat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24683.txt.

Introduction

Dayeon Ki, Kevin Duh, and Marine Carpuat test what kind of quality feedback helps people trust machine translation wisely in a 2025 paper. Monolingual users often must decide whether to share a translation they cannot read. Feedback could calibrate that trust, but its best form was unknown.

Users first judged translations unaided, then with one of four aids: error highlights, written explanations from a language model, back-translation, or a question-answer table cross-checking content. Everything except error highlights improved decision accuracy and appropriate reliance. The question-answer tables won clearly, beating even direct explanations, while earning the best marks for helpfulness and trust and the lowest mental effort. The authors argue showing comparisons beats issuing verdicts.

Catalog record

CREATORDayeon Ki Kevin Duh Marine Carpuat
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24683

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translation Dayeon Ki Kevin Duh Marine Carpuat Abstract As people increasingly use AI systems in work and daily life, feedback mechanisms that help them use AI responsibly are urgently needed, particularly in settings where users are not equipped to assess the quality of AI predictions. We study a realistic Machine Translation (MT) scenario where monolingual users decide whether to share an MT output, first without and then with quality feedback. We compare four types of quality feedback: explicit feedback that directly give users an assessment of translation quality using 1) error highlights and 2) LLM explanations, and implicit feedback that helps users compare MT inputs and outputs through 3) backtranslation and 4) question–answer (QA) tables. We find that all feedback types, except error highlights, significantly improve both decision accuracy and appropriate reliance. Notably, implicit feedback, especially QA tables, yields significantly greater gains than explicit feedback in terms of decision accuracy, appropriate reliance, and user perceptions -- receiving the highest ratings for helpfulness and trust, and the lowest for mental burden. 1 1 1 We release our code at https://github.com/dayeonki/mt_quality_feedback . 1 Introduction Figure 1: In our human study, each English-speaking monolingual participant reviews a sequence of 20 decision-making examples. Each example is shown in a two-step process: ➊ Independent decision-making: Participants first make judgments based solely on the English source and its Spanish MT output and ➋ AI-Assisted decision-making: They then reassess the same example with one of four randomly assigned feedback types . For each step, they respond to two questions: (1) Shareability: To the best of your knowledge, is the Spanish translation good enough to safely share with your Spanish-speaking neighbor? and (2) Confidence: How confident are you in your assessment? Artificial Intelligence (AI) are increasingly deployed to support human decision-making across a wide range of domains ( Buçinca et al. 2021 ; Dastin 2022 ; Ma et al. 2024 ) . As these systems are adopted by the general public, there is a growing need for feedback mechanisms that help users construct their own functional explanations – reasoning grounded in the goals and consequences of an AI output to determine how and when to rely on it for safe and effective use ( Lombrozo and Wilkenfeld 2019 ; Schoeffer et al. 2024 ) . Prior work has evaluated various forms of feedback through human studies, such as error highlights ( Khashabi et al. 2018 ; Carton et al. 2020 ) or free-text explanations ( Bussone et al. 2015 ; Bansal et al. 2021 ; Buçinca et al. 2021 ) . However, many human-centered studies evaluating the impact of feedback in real-world application settings are still needed. Although designing such studies is challenging, as it requires accounting for the knowledge and assumptions people bring to decision-making tasks ( Lage et al. 2019 ; Wiegreffe and Marasovic 2021 ) , it remains essential to adopt a human-centered evaluation since these feedback mechanisms are ultimately intended to support human users ( Boyd-Graber et al. 2022 ; Zhu et al. 2024 ) . To address this, and in line with prior work that empirically investigates real-world use cases in other domains ( Hong et al. 2020 ; Bhatt et al. 2020 ; Liao et al. 2020 ) , we evaluate AI feedback for Machine Translation (MT), where many users critically need support because they lack the language proficiency needed to evaluate MT outputs. Imagine a situation during the COVID-19 pandemic: You regularly read official guidelines in English, but your Spanish-speaking neighbor cannot access this information. You turn to MT to share this information – but as an English monolingual, how can you determine whether the Spanish MT output is accurate enough to safely share, or if it contains critical errors risking misinformation? This is a practical yet challenging scenario for monolingual users, who lack both source language proficiency ( Bowker and Ciro 2019 ; Liebling et al. 2020 ) and domain expertise ( Nourani et al. 2020 ; Lee and Chew 2023 ) to reliably evaluate MT quality. Prior studies have proposed quality feedback mechanisms to support MT decision-making, such as paraphrases, Quality Estimation (QE) scores, and backtranslation, but findings on their impact on user confidence and decision accuracy remain mixed ( Zouhar et al. 2021 ; Mehandru et al. 2023 ) . Building on this, we provide a more comprehensive assessment of which feedback types best support users in forming functional explanations to make reliable MT decisions. We evaluate four types of quality feedback, grouped by their mode of explicitness: 1) Explicit quality assessments of MT output (error highlights and LLM explanation); and 2) Implicit assessments that support input/output comparison (backtranslation and QA table). We conduct a human study with 73 English-speaking monolingual participants, where they are asked to decide whether Spanish MT outputs are safe to share with a hypothetical Spanish-speaking neighbor. For each of 20 examples, participants first make a decision without feedback (Independent step), then reassess the same example with a randomly assigned form of feedback (AI-Assisted step), as illustrated in Figure 1 . Our findings reveal that all feedback types except error highlights significantly improve both decision accuracy and appropriate reliance (§ 4.1 ). Implicit feedback generally outperforms explicit types: backtranslation yields significantly higher appropriate reliance than error highlights, and QA table leads to significantly higher gains in both metrics than both explicit feedback types (§ 4.2 ). While explicit feedback prompts more decision changes, it also results in higher over-reliance, which are cases where participants change from correct to incorrect decisions after viewing feedback (§ 4.3 ). We also find that participants are better at recognizing good translations than identifying problematic ones (§ 5.1 ). In terms of self-reported perceptions on mental burden, helpfulness, and trust, QA table feedback consistently receives the best ratings (§ 5.2 ). Finally, we present participants’ responses to identify which aspects of each quality feedback they found to be helpful (§ 5.3 ). Together, these results highlight the value of quality feedback that supports users’ implicit interpretation of MT outputs rather than explicitly telling them what to do, supporting a more user-driven process for making reliable decisions. 2 Background & Research Questions 2.1 Reliance on AI Systems A growing body of work has examined the nature of human reliance on AI systems ( Lai et al. 2023 ) , particularly in scenarios involving risk and uncertainty ( Jacovi et al. 2021 ) . While the overarching goal is to design trustworthy AI, studying trust is complex and multifaceted. To operationalize this, prior works have developed methods to study human behavior when using AI systems with a focus on reliance ( de Fine Licht and Brülde 2021 ) , defined as the “decision to follow someone’s recommendation” ( Vereschak et al. 2021 ) . Various metrics have been proposed to assess the degree of user reliance, including agreement percentage (how often a user agrees with the AI prediction) ( He et al. 2023a ) , confidence-weighted accuracy (decision accuracy weighted by user confidence) ( Mehandru et al. 2023 ) , and switch percentage (how often a user changes their decision after seeing AI feedback) ( Schmitt et al.

Read the complete text