Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translation

Di Wu Seth Aycock Christof Monz Language Technology LabUniversity of Amsterdam {d.wu, s.aycock, c.monz}@uva.nl · 2025

Read Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translation by Di Wu Seth Aycock Christof Monz Language Technology LabUniversity of Amsterdam {d.wu, s.aycock, c.monz}@uva.nl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04521.txt.

Introduction

Di Wu, Seth Aycock, and Christof Monz test whether human-style reasoning helps machine translation in a 2025 computing paper. A popular recipe asks language models to translate step by step, drafting and refining like a person. That recipe topped a recent world contest. The authors doubt the steps themselves deserve the credit, and they run two simple experiments to check.

Asking the model to simply translate again and polish its own output beat the step-by-step recipe. Following the steps faithfully helped in some ways and hurt in others. The gains, they argue, come from the extra pass, not from copying human habits. The result warns builders not to assume that what helps people helps models too.

Catalog record

CREATORDi Wu Seth Aycock Christof Monz Language Technology LabUniversity of Amsterdam {d.wu, s.aycock, c.monz}@uva.nl
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04521

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translation Di Wu Seth Aycock Christof Monz Language Technology LabUniversity of Amsterdam {d.wu, s.aycock, c.monz}@uva.nl Abstract Large Language Models (LLMs) demonstrate strong reasoning capabilities for many tasks, often by explicitly decomposing the task via Chain-of-Thought (CoT) reasoning. Recent work on LLM-based translation designs hand-crafted prompts to decompose translation, or trains models to incorporate intermediate steps. Translating Step-by-step ( Briakou et al. 2024 ) , for instance, introduces a multi-step prompt with decomposition and refinement of translation with LLMs, which achieved state-of-the-art results on WMT24. In this work, we scrutinise this strategy’s effectiveness. Empirically, we find no clear evidence that performance gains stem from explicitly decomposing the translation process, at least for the models on test; and we show that simply prompting LLMs to “translate again” yields even better results than human-like step-by-step prompting. Our analysis does not rule out the role of reasoning, but instead invites future work exploring the factors for CoT’s effectiveness in the context of translation. \DeclareCaptionType example[Example][List of examples] \DeclareCaptionType prompt[Prompt][List of prompts] \DeclareCaptionType equ[Equation][List of equations] 0 0 footnotetext: We plan to release the 223k segment and document translations in 8 language pairs from 3 models across 4 steps at [anonymised] to enable further research into the effects of decomposition on translation quality. 1 Introduction Large Language Models (LLMs) exhibit strong reasoning capabilities, often characterized by a lengthy, step-by-step decomposition of the question prior to generating the answer—known as Chain-of-Thought (CoT) ( Wei et al. 2022 ) —along with possible attempts and revisions of the answer, referred to as self-refinement ( Madaan et al. 2023 ; Chen et al. 2023 ; Pan et al. 2024 ) . Both processes resemble human behaviour when tackling complex problems such as coding and mathematics, and have been shown to be effective for such tasks. Driven by recent advancements in LLMs’ reasoning capabilities, a trend has developed in improving translation quality through a human-like decomposition–translation–refinement paradigm. Here, the source text is decomposed into different aspects including meanings, topics, idiomatic expressions etc., followed by translation drafting and refinement based on these aspects, before generating the final translation. Some recent works explore pre-translation decomposition, focusing on keywords ( He et al. 2024 ) or idiomatic expressions ( Li et al. 2024 ) , aided by external resources. Others address post-translation refinement, guided by external translation quality assessment ( Huang et al. 2024 ; Ki and Carpuat 2024 ) or explicit self-evaluation ( Feng et al. 2025 ) . Refinement can be applied iteratively ( Chen et al. 2023 ; Xu et al. 2024 ) , and is particularly effective for long document-level translation ( Wu et al. 2024 ) . A key work by Briakou et al. 2024 combines pre- and post-translation processes via a fixed 4-step prompting strategy—decomposition (or research), drafting, refinement, and proofreading. Their method shows progressive improvements for long-form translation, achieving state-of-the-art results on WMT24. Figure 1: Schematic of prompting frameworks for Step-by-step translation with decomposition (above) and multi-pass Translate again (our method, below), with user prompts and model outputs shown for each step. Experiment 1 ( exp . 1) compares translation and refinement outcomes with and without the decomposition step across metrics, input types, and models. Experiment 2 ( exp . 2) traces back evidence to assess whether accurately following decomposition improves translation. Full prompts for both settings are provided in Appendix C . While these studies show performance gains over direct translation in some settings, the generalizability of human-like multi-pass prompting across models and input types remains unclear. Further, most lack an explicit examination or quantitative analysis of the underlying mechanisms behind these gains. To address these points, we design two simple experiments comparing against the current best practice ( Briakou et al. 2024 ) to answer: • Does decomposition positively impact translation quality, across models and input types? • How faithful are translations to their decomposition, and does faithfulness lead to improved translation quality? Our two simple experiments find that: (1) most gains come from self-refinement, while decomposition has limited—and sometimes negative—effects, depending largely on the LLM and input type; and (2) decomposition clearly influences translation behaviour, but strict faithfulness to the decomposition does not necessarily improve translation quality. Given the findings, we encourage the research community to evaluate alternative explanations and reconsider the necessity of human-like decomposition when engaging the reasoning capabilities of LLMs for translation. At a minimum, future studies should consider incorporating a CoT-free refinement strategy—such as the simple ‘please translate again’ prompt used here—as a baseline, given its demonstrated effectiveness and efficiency. 2 Translation Decomposition and Refinement (a) GPT—Segment (b) GPT—Document (c) GPT—Segment (d) GPT—Document (e) Gemini—Segment (f) Gemini—Document (g) Gemini—Segment (h) Gemini—Document Figure 2: Step-by-step vs. Translate again results in COMET-22 and CometKiwi-XL for GPT-4o-mini (top) and Gemini-2.0-Flash (bottom), for segment and document-level translation. Note Steps 2–4 iteratively call the LLM to translate again, and Step 4 is a proofreading step; see Fig. 1 for an illustration and Appendix C for full prompts. Translation by human translators is commonly divided into three phases: pre-drafting, drafting, and post-drafting ( Mossop 2013 ) . First, the translator familiarises themselves with the source, consisting of comprehension and planning; next, a full draft translation is written, optionally with the help of external resources; then the translator reviews and revises the draft translation. Briakou et al. 2024 partially replicate this process with their 4-step prompting process, splitting the final step into refinement and proofreading. Formally, given a language model p_{\theta} and a source text x to be translated, the output can be viewed as a sample O\sim p_{\theta}(\cdot\mid I(x)) , where I(x) is a prompt that may include x as a component. Multi-step prompting for translation is a sequential process in which the outputs of previous steps are fed into the next prompt. For instance, the decomposition–translation–refinement workflow (Figure 1 , top) can be formalized as: \displaystyle O_{d} \displaystyle\sim p_{\theta}(\cdot\mid I_{d}(x)), \displaystyle O_{t} \displaystyle\sim p_{\theta}(\cdot\mid I_{d}(x),O_{d},I_{t}(x)), \displaystyle O_{f} \displaystyle\sim p_{\theta}(\cdot\mid I_{d}(x),O_{d},I_{t}(x),O_{t},I_{f}(x)). Here, I_{d} , I_{t} , and I_{f} denote the prompts for the decomposition, translation, and refinement steps, respectively, and O_{d} , O_{t} , and O_{f} are their corresponding outputs. This study investigates the impact of human-like decomposition ( O_{d} ) on translation quality ( O_{t} ) and final refinement output ( O_{f} ) under varying conditions: (i) model differences ( \theta ), (ii) sentence- vs. document-level source inputs x , and (iii) the presence or absence of O_{d} (see Section 4 ). We also explicitly verify whether faithfully following the decomposition generally leads to improved translation (see Section 5 ). 3 Experimental Setup Models. We use GPT-4o-mini ( OpenAI 2024 ) and Gemini-2.0-Flash ( Google 2024 ) , as performant and cost-effective closed-source LLMs that demonstrate strong reasoning capabilities. Data. We use the WMT24++ ( Deutsch et al. 2025 ) dataset as our test set, because a) the dataset was released later than the LLMs we used here, ensuring no data leakage issues, and b) the translation references in WMT24++ are human-written and subsequently post-edited by professional translators, ensuring the highest possible data quality. In this study, we use the post-edited version. We select 8 language pairs ( en→cs,de,fr,he,ja,ru,uk,zh ) to cover varying

Read the complete text