Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation
Shaharukh Khan, Ayush Tarun, Ali Faraz, Palash Kamble, Vivek Dahiya, Praveen Pokala *, Ashish Kulkarni *, Chandra Khatri *, Abhinav Ravi * and Shubham Agarwal * Krutrim AI, Bangalore, India * Senior Contributors · 2025
Read Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation by Shaharukh Khan, Ayush Tarun, Ali Faraz, Palash Kamble, Vivek Dahiya, Praveen Pokala *, Ashish Kulkarni *, Chandra Khatri *, Abhinav Ravi * and Shubham Agarwal * Krutrim AI, Bangalore, India * Senior Contributors (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20420.txt.
Introduction
Shaharukh Khan and a ten-author team wrote a 2025 paper describing Chitranuvad, their system for translating English into Indic languages with images in the loop. Multimodal translation uses a picture to disambiguate text, a help for captions and low-resource languages. Their entry competed in a shared task at the Workshop on Asian Translation 2024.
The system pairs a multilingual language model with a vision transformer. Image features are extracted, then projected through an adapter into the language model's space as visual tokens. Translation is generated step by step from text and image together. The team entered all three tracks: captioning, text-only, and multimodal translation into Hindi, Bengali, and Malayalam. They took state-of-the-art results for Hindi on every track's challenge set. Performance stayed competitive for the other two languages.
Catalog record
| CREATOR | Shaharukh Khan, Ayush Tarun, Ali Faraz, Palash Kamble, Vivek Dahiya, Praveen Pokala *, Ashish Kulkarni *, Chandra Khatri *, Abhinav Ravi * and Shubham Agarwal * Krutrim AI, Bangalore, India * Senior Contributors |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-20420
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation Shaharukh Khan, Ayush Tarun, Ali Faraz, Palash Kamble, Vivek Dahiya, Praveen Pokala *, Ashish Kulkarni *, Chandra Khatri *, Abhinav Ravi * and Shubham Agarwal * Krutrim AI, Bangalore, India * Senior Contributors Abstract In this work, we provide the system description of our submission as part of the English-to-Lowres Multimodal Translation Task at the Workshop on Asian Translation (WAT2024). We introduce Chitranuvad, a multimodal model that effectively integrates Multilingual LLM and a vision module for Multimodal Translation. Our method uses a ViT image encoder to extract visual representations as visual token embeddings which are projected to the LLM space by an adapter layer and generates translation in an autoregressive fashion. We participated in all the three tracks (Image Captioning, Text-only and Multimodal translation tasks) for Indic languages (ie. English translation to Hindi, Bengali and Malyalam) and achieved SOTA results for Hindi in all of them on the Challenge set while remaining competitive for the other languages in the shared task. 1 Introduction Recently, there has been an increased interest in Multimodal Machine Translation (MMT) task Calixto and Liu 2017 ; Delbrouck and Dupont 2017 ; Elliott and Kádár 2017 ; Yao and Wan 2020 which involves translation between language pairs, incorporating other modalities (like images) as an auxiliary information. The visual cues act as ‘symbol grounding’ Fodor 1975 ; Harnad 1990 ; Harnad 2003 ; Harnad 2005 , helping to resolve ambiguities in language ( Rainie et al. 2012 ; Hu et al. 2014 ; Specia et al. 2016 ; van Miltenburg et al. 2019 ; Caglayan et al. 2020 ) by learning to connect language and perception Mooney 2008 ; Bisk et al. 2020 . For example, in order to correctly translate the word court in Figure 1 , the model has to infer from the image that the statement is about tennis court and not the court as government institution. Prior works mostly focused on translation from English to European languages Elliott et al. 2016 ; Specia et al. 2016 while the Indic languages remain largely unexplored, with an exception of the MMT shared task at the Workshop on Asian Translation (WAT) Nakazawa et al. 2019 ; Nakazawa et al. 2020 ; Nakazawa et al. 2021 ; Nakazawa et al. 2022 ; Nakazawa et al. 2023 . The English-to-Lowres Multimodal Translation Task at WAT-2024 targets the MMT task for three Indic medium-to-low-resource languages (Hindi, Bengali, Malayalam) and a low-resource African language Hausa. To assess the importance of the image modality, the task has been decoupled into three tracks: 1). Text-only translation where the source image is not used, 2). Image Captioning where English source text is not used and 3). Multimodal translation which uses both the image and the text. We participated in all the three tracks for Indic languages only (Hindi, Bengali, Malayalam) under a non-constrained and proprietary multi-lingual and multimodal Large Language Model (LLM): Chitranuvad 1 1 1 Chitranuvad literally means Image Translate in Hindi . Figure 1: Multimodal Machine Translation task as part of English-to-lowres track where the source sentence is translated to multiple Indic languages (Hindi, Bengali, Malayalam) grounded in the image. Meaning of words like "court" and "right" in the translations can vary significantly depending on the visual context. In this paper, we provide a description of our multimodal LLM where we leverage a multi-lingual LLM backbone Krutrim Kallappa et al. 2024 , coupled with a visual image encoder. Our contributions could thus be summarized as follows: • We introduce Chitranuvad, a Large Multimodal model, adapted for multi-lingual translation, which leverages images and language modalities to provide an image grounded translation of the English sentence in the target Indic languages. • We showcase the effectiveness of task specific finetuning on the Visual Genome translation datasets and achieve SOTA performance. • We evaluate Chitranuvad and prior baselines on the English-to-Lowres Multimodal Translation Task and demonstrate the ability of our model to perform grounded translation, using different training strategies and ablations. The rest of the paper is organized as: Section 2 presents related research on multimodal machine translation while Section 3 explains our Chitranuvad model recipe in detail. We present the datasets used in Section 4 , followed by experimental findings in Section 5 and conclusion in Section 6 . Figure 2: Chitranuvad model architecture with the three stage training pipeline described in Section 3 . 2 Related Work Early Neural Machine Translation (NMT) and Image captioning systems Show 2015 ; Gao et al. 2018 were based on Recurrent Neural Networks (RNNs) and their variants Cho et al. 2014 ; Sutskever et al. 2014 ; Cho 2014 ; Hochreiter and Schmidhuber 1997 , often incorporating attention mechanisms Bahdanau et al. 2014 . The seminal work of transformers Vaswani 2017 paved the way for the development of high-quality image captioning Chen et al. 2021 as well as translation systems Lewis 2019 , even for low-resource languages Dabre et al. 2021 ; Gala et al. 2023a . Multimodal Machine Translation (MMT) systems witnessed a similar shift in their approrach Caglayan et al. 2016 ; Yao and Wan 2020 ; Guo et al. 2023 . Prior submissions to the MMT task at Workshop on Asian Translation Gain et al. 2021 ; Gupta et al. 2021 ; Parida et al. 2022 ; Dash et al. 2023 ; Shahid et al. 2023 also fall in this category. The next generation of Multimodal LLMs Lu et al. 2024a ; Laurençon et al. 2024 ; Tong et al. 2024 ; Xue et al. 2024 can handle a variety of complex tasks, including machine translation and captioning, by utilizing cutting-edge architectures as an unified general purpose agent. These models often rely on pre-trained LLMs, with an exception of few, which train the models from scratch Team 2024 ; Lu et al. 2024b . Most of these Vision Language Models (VLMs) follow the architecture of Liu et al. 2023a where a CLIP Radford et al. 2021 or a similar encoder is used to encode the image and projected into LLM’s representation space using an adapter layer. Notably, Wang et al. 2023 offers a departure from conventional architectures by using distinct matrices and Feed Forward Networks for image modalities. Recent developments replace the image encoder with SigLIP Zhai et al. 2023a and the single-layer MLP projector with attention-based pooling Laurençon et al. 2024 . Advanced backbone LLMs Brown et al. 2020 ; Touvron et al. 2023 ; Achiam et al. 2023 ; Team et al. 2023 ; Jiang et al. 2024 ; Team et al. 2024 however have a primary focus for English and European languages. There have been relatively few LLMs for Indic languages, such as Airavata Gala et al. 2024 , Navarsa Labs 2023 , Kannada LLaMA, Tamil LLaMA Balachandran 2023 , Odia LLaMA Kohli et al. 2023 , to name a few. However, most of these LLMs are an extension and finetuned version of LLaMA/Gemma for Indic languages, which don’t fully capture the nuances of the language. This could be attributed to the fact that Indic languages are under-represented in Common Crawl (which majorly forms the training corpus of LLMs), despite India constituting 18% of the global population. Hindi, for example, does not show-up