Exploring In-Image Machine Translation with Real-World Background

Yanzhi Tian Zeming Liu Zhengyang Liu Yuhang Guo · 2025

Read Exploring In-Image Machine Translation with Real-World Background by Yanzhi Tian Zeming Liu Zhengyang Liu Yuhang Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15282.txt.

Introduction

Yanzhi Tian and co-authors tackle a hard translation task in this 2025 paper: text that sits inside a photo or sign, with a real-world background behind it. Older work used simple images, such as black letters on a white page. Their dataset instead pairs subtitle text with real scenes, which is much closer to daily use.

Their model, called DebackX, first splits the source image into background and text layers. It then translates the text layer alone and blends the result back onto the background. In tests, the model beat older systems on both translation quality and how natural the final image looked.

Catalog record

CREATORYanzhi Tian Zeming Liu Zhengyang Liu Yuhang Guo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15282

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring In-Image Machine Translation with Real-World Background Yanzhi Tian Zeming Liu Zhengyang Liu Yuhang Guo Abstract In-Image Machine Translation (IIMT) aims to translate texts within images from one language to another. Previous research on IIMT was primarily conducted on simplified scenarios such as images of one-line text with black font in white backgrounds, which is far from reality and impractical for applications in the real world. To make IIMT research practically valuable, it is essential to consider a complex scenario where the text backgrounds are derived from real-world images. To facilitate research of complex scenario IIMT, we design an IIMT dataset that includes subtitle text with real-world background. However previous IIMT models perform inadequately in complex scenarios. To address the issue, we propose the DebackX model, which separates the background and text-image from the source image, performs translation on text-image directly, and fuses the translated text-image with the background, to generate the target image. Experimental results show that our model achieves improvements in both translation quality and visual effect. 1 1 1 Code and dataset: https://github.com/BITHLP/DebackX 1 Introduction In-Image Machine Translation (IIMT) aims to translate texts within images from one language to another, and related technologies are widely used in translation applications Mansimov et al. (2020) ; Tian et al. (2023) ; Lan et al. (2024) . IIMT performs translation on visual modality, helping people understand context in images directly, which is different from the text-to-text machine translation. Figure 1: Illustration of simplified and complex scenario IIMT. Previous research mainly focuses on simplified scenario IIMT, which is far from reality. Exploring complex IIMT scenarios that are much closer to reality is necessary. We find that the current IIMT model cannot fully handle the complex scenario, since the translation lacks an integrated background and fails to maintain font consistency, leading to poor visual effect. IIMT is a challenging machine translation task, and previous research mainly focuses on simplified scenarios as shown in Figure 1 (a). Tian et al. (2023) translate images containing texts with black font and white background. Lan et al. (2024) construct images with a single color background. However, the aforementioned research overly simplifies the IIMT task, ignoring the fact that most of the text in the real world appears with a complex background, such as video subtitles, as shown in Figure 1 (b). The limitation makes previous approaches impractical for applications in the real world, highlighting the need for a scenario that is closer to reality. To overcome the limitation and make IIMT research practical, we focus on a more realistic IIMT task, namely complex scenario IIMT, building a new IIMT dataset where the images contain text integrated with real-world background. Compared to the simplified IIMT task that hardly aligns with reality, our designed task better reflects real world. On the complex scenario dataset, we experiment with a feasible OCR-NMT-Render model, which primarily consists of four steps: (1) Recognizing the source language texts with the Optical Character Recognition (OCR) model; (2) Translating the source texts recognized by the OCR model with the Neural Machine Translation (NMT) model; (3) Removing the text areas in the source image to erase the texts; (4) Rendering the translation results into the corresponding position of the image. The above framework has the following drawbacks. Firstly, the cascading OCR and NMT models have the risk of error propagation, leading to the decreasing translation quality, which is demonstrated in the previous works Tian et al. (2023) ; Lan et al. (2024) . Secondly, removing text areas in step (3) affects the integrity of the source images, decreasing the visual effect of rendering results. Thirdly, OCR models generally only give recognized text results, leading to the loss of font information during the cascading process, and it is hard to maintain consistency for font styles of texts between the source images and target images. The comparison between the ideal translation result and the translation produced by the OCR-NMT-Render model is shown in Figure 1 (b). To address the drawbacks, we design the DebackX model for complex scenario IIMT, consisting of three components: (i) A Text-Image Background Separation model is used to generate the background and text-image from the source image. Processing the background and text-image separately, our model ensures the integrity of the background. (ii) An Image Translation model is used to model the transformation from the text-image of the source language to the text-image of the target language. The direct translation mitigates the decreasing translation quality caused by error propagation between OCR and NMT and helps maintain font consistency between the target images and source images. (iii) A Text-Image Background Fusion model is used to fuse the background and target text-image to generate the final output. The main contributions of this paper are as follows: • We propose a complex scenario IIMT, which requires translating text within real-world backgrounds to align closely with reality. • We build a new dataset IIMT30k containing backgrounds derived from real-world images for complex scenario IIMT. Existing IIMT models, including GPT-4o, fail to achieve both high-quality translation and visual effects. • We propose the DebackX model for complex scenario IIMT, processing the background and text of the image separately, achieving both better translation quality and visual effect compared to current IIMT models. 2 Related Work Text-Image Translation. Conventional research on Neural Machine Translation usually focuses on text and speech modality Vaswani et al. (2023) ; Inaguma et al. (2023) . There is also research focusing on translating the text in the image, which is an image-to-text multi-modality translation task, referred to Text-Image Translation (TIT). Ma et al. (2022) construct a synthetic dataset and employ multi-task learning in the training of end-to-end TIT model. Lan et al. (2023) annotate the OCRMT30K dataset, and propose an OCR-NMT cascade TIT model, introducing visual information with a multimodal codebook into the NMT model. Ma et al. (2023a) propose a cross-model cross-lingual interactive model for TIT with weighted interactive attention and hierarchical interactive attention. Zhu et al. (2023) design a shared encoder-decoder backbone with vision-text representation aligner and cross-model regularizer. In-Image Machine Translation. All of the above works only generate the translated texts but do not generate the images with translation. Mansimov et al. (2020) explore the In-Image Machine Translation (IIMT) task, aiming to transform images containing texts from one language to another, which is closer to the real-world applications. Both the input and output of IIMT are entirely based on the image modality, significantly different from other machine translation tasks. Tian et al. (2023) construct a dataset containing images of one-line texts with black font and white background, improving the translation quality of IIMT with a segmented pixel transformer. Lan et al. (2024) adopt the ViT-VQGAN to IIMT with OCR and TIT multi-task learning, constructing a dataset with a single color as a background with multi-line texts. Qian et al. (2024) utilize Qwen and AnyText models to translate images with short text on a real dataset but do not release their dataset publicly. Text-within-Image Generation The research of text-within-image generation aims to generate clear and readable text in the images, which is generally necessary to incorporate textual priors, such as text position and image with rendered text Ma et al. (2023b) ; Tuo et al.

Read the complete text