Tower+ : Bridging Generality and Translation Specialization in Multilingual LLMs

Ricardo Rei, Nuno M. Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, André F. T. Martins · 2025

Read Tower+ : Bridging Generality and Translation Specialization in Multilingual LLMs by Ricardo Rei, Nuno M. Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, André F. T. Martins (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17080.txt.

Introduction

Ricardo Rei and co-authors build translation models in 2025 that refuse to forget everything else. Tuning a model hard for translation usually dulls its chat, reasoning, and instruction skills, a poor trade for real use. Their suite, Tower+, aims to hold both: top translation and broad multilingual skill.

The recipe stacks four stages on an earlier translation model: more reading, guided tuning, preference polish, and reward-based training with checkable answers. At each stage the data mix feeds both translation and general skills like code, math, and following orders, they report. Sizes run from two billion parameters upward. Holding the frontier between specialist and generalist is the point: one model a service can both translate with and talk to.

Catalog record

CREATORRicardo Rei, Nuno M. Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, André F. T. Martins
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-17080

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tower+ : Bridging Generality and Translation Specialization in Multilingual LLMs Ricardo Rei, Nuno M. Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, André F. T. Martins Abstract Fine-tuning pretrained LLMs has been shown to be an effective strategy for reaching state-of-the-art performance on specific tasks like machine translation. However, this process of adaptation often implies sacrificing general-purpose capabilities, such as conversational reasoning and instruction-following, hampering the utility of the system in real-world applications that require a mixture of skills. In this paper, we introduce Tower+ , a suite of models designed to deliver strong performance across both translation and multilingual general-purpose text capabilities. We achieve a Pareto frontier between translation specialization and multilingual general-purpose capabilities by introducing a novel training recipe that builds on Tower ( Alves et al. 2024 ) , comprising continued pretraining, supervised fine-tuning, preference optimization, and reinforcement learning with verifiable rewards. At each stage of training, we carefully generate and curate data to strengthen performance on translation as well as general-purpose tasks involving code generation, mathematics problem solving, and general instruction-following. We develop models at multiple scales: 2B, 9B, and 72B. Our smaller models often outperform larger general-purpose open-weight and proprietary LLMs (e.g., Llama 3.3 70B, GPT-4o). Our largest model delivers best-in-class translation performance for high-resource languages and top results in multilingual Arena Hard evaluations and in IF-MT, a benchmark we introduce for evaluating both translation and instruction-following. Our findings highlight that it is possible to rival frontier models in general capabilities, while optimizing for specific business domains, such as translation and localization. 1 Introduction 80 81 82 83 84 85 86 87 88 0 10 20 30 40 50 60 70 Tower v2 Llama 3.3 70B Qwen 2.5 72B Gemma 2 9B Gemma 2 27B GemmaX 9B ALMA-R 13B GPT-4o Claude Sonnet 3.7 Tower+ 9B Tower+ 72B Translation Performance ( xComet-xxl on WMT24++ ) General Capabilities Performance (M-ArenaHard) Figure 1: Translation and general capabilities performance of state-of-the-art translation-specific (circles) and general-purpose LLMs (logos) of varying sizes (from 9B to 72B). For their size, Tower+ models outperform or match open-weight models on both axes, and Tower+ 72B is competitive to state-of-the-art closed models. We omit 2B models for better visualization; we show detailed results in Table 1 . Gray lines connect each Tower+ model to their respective “instruction-tuned” counterpart. Large Language Models (LLMs) are rapidly emerging as the de facto solution for multilingual machine translation. Recent studies and shared tasks have shown that state-of-the-art proprietary LLMs, such as GPT-4o and Claude 3.7 are currently state-of-the-art in translation quality Kocmi et al. 2023 ; Kocmi et al. 2024a ; Deutsch et al. 2025 . At the same time, several studies have shown that open-weight LLMs can be effectively adapted for machine translation, reaching parity with or even surpassing the translation quality of leading proprietary models Alves et al. 2024 ; Rei et al. 2024 ; Xu et al. 2025 ; Cui et al. 2025a . However, these task-specific adaptations often come at the cost of general-purpose capabilities 1 1 1 The term “general-purpose capabilities” refers to the real-world utility of language models in handling queries that require core knowledge, instruction-following, and conversational reasoning ( Zheng et al. 2023 ) such as instruction following and conversational reasoning. For example, Tower v2 , the best-performing system in the WMT24 shared task, ranked first in 8 out of 11 language pairs Kocmi et al. 2024a , yet underperforms models like GPT-4o or Claude 3 on general chat evaluations. This tradeoff is illustrated in Figure 1 , which shows that translation-specialized models tend to fall short on the Pareto frontier that balances translation quality and general-purpose utility. In addition, the degradation of instruction-following capabilities can hinder the ability to handle complex translation scenarios, such as adhering to a terminology or formatting rules. Indeed, translation-specific systems greatly underperform general-purpose models on IF-MT, a novel benchmark we introduce for evaluating both translation and instruction-following. To address this challenge, we investigate how to develop state-of-the-art translation models without compromising performance on general chat benchmarks. Our post-training pipeline builds on the framework introduced in earlier Tower models Alves et al. 2024 ; Rei et al. 2024 , but introduces several important methodological improvements. We begin with continued pretraining (CPT) on a curated mixture of monolingual and parallel data to enhance multilingual fluency and translation performance. Unlike previous versions, the CPT stage now also includes a small fraction (1%) of high-quality instruction-like data, which helps preserve general capabilities during early specialization. For supervised fine-tuning (SFT), we refine the data mixture and significantly rebalance task weights: while translation remains a key component, it now accounts for only 22% of the SFT corpus, with the remainder covering instruction-following tasks such as mathematics, code generation, and question answering. In addition, we improve the quality of SFT responses through automated refinement using open-weight models, resulting in better quality answers. Finally, we introduce a new preference optimization stage using Weighted Preference Optimization ( Zhou et al. 2024 , WPO) , complemented by Group Relative Policy Optimization ( Shao et al. 2024 , GRPO) and verifiable reward models (RLVR), to further align the model with high-quality outputs across tasks. We evaluate our models on both general chat benchmarks and translation tasks. Crucially, we analyze the contribution of each stage of the training pipeline in Section 3.3 , and assess the role of the base model in balancing general-purpose and translation-specific capabilities in Section 3.4 . Our contributions are as follows: • We present, to the best of our knowledge, the first systematic study on balancing translation quality and general-purpose capabilities in open-weight LLMs. While most prior work ( Alves et al. 2024 ; Xu et al. 2024a ; Rei et al. 2024 ; Cui et al. 2025a ) has focused solely on maximizing translation performance, our approach explicitly targets a broader trade-off. • We introduce a post-training pipeline that integrates diverse multilingual signals without compromising general chat abilities. Our approach can serve as a blueprint for adapting LLMs to domain- or task-specific business use cases while preserving general capabilities. • We introduce and release IF-MT, 2 2 2 We make IF-MT and all Tower+ models available on Huggingface . a novel benchmark for evaluating both translation and instruction-following capabilities on two langauge pairs (English \rightarrow Spanish and English \rightarrow Chinese). • We release Tower+ , 2 a suite of models that demonstrate strong performance across translation, general capabilities, and a benchmark that mixes the two. We match or exceed the translation quality of prior Tower models and GPT-4o-1120 , while also surpassing general-purpose open-weight models like Llama-3.3 70B and Qwen2.5 72B on M-ArenaHard. Figure 2: Process for creating and curating data for our final dataset for SFT. 2 Tower Post-Training Our refined post-training pipeline consists of four stages: Continued Pretraining (CPT), supervised fine-tuning (SFT) (as in Tower-v2 , Rei et al. 2024 ), preference optimization using WPO Zhou et al. 2024 , and finally, reinforcement learning with verifiable rewards using GRPO Shao et al. 2024 . In this section, we describe each stage in detail and explain how translation signals are incorporated throughout the pipeline. 2.1 Continued Pretraining The continued pretraining phase leverages monolingual, parallel, and general-purpose instruction-following data. Similarly to

Read the complete text