Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages

Swastik R · 2026

Read Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages by Swastik R (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26742.txt.

Introduction

R presents a cross-lingual audit of visual reasoning in eight vision-language models across Indian languages. Reasoning tests are usually run in English, which can hide sharp drops elsewhere. The audit moves 980 questions from MathVista, ScienceQA, and MMMU into Hindi, Tamil, Telugu, Bengali, Kannada, and Marathi. Accuracy falls by 9.8 to 25 points when English is swapped out. Dravidian languages lose up to 13.2 points more than Indo-Aryan ones, and chain-of-thought prompts hurt Bengali and Kannada rather than helping.

Catalog record

CREATORSwastik R
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26617
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26742

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages Swastik R Abstract Vision-language models score well on mathematical, scientific, and spatial reasoning benchmarks, yet these evaluations are overwhelmingly English. I present the first cross-lingual visual reasoning audit for Indian languages. 980 questions from MathVista, ScienceQA, and MMMU are translated into Hindi, Tamil, Telugu, Bengali, Kannada, and Marathi using IndicTrans2, with Gemini 2.0 Flash cross-verification on 50 samples per language (inter-translator agreement 0.79–0.84). Eight VLMs, from 7B open-source models to GPT-4o, are evaluated across all seven languages, yielding 68,600 inference records that include text-only and chain-of-thought ablations. I find accuracy drops of 9.8–25 percentage points when switching from English to an Indian language, with Dravidian languages suffering up to 13.2 pp more than Indo-Aryan. Chain-of-thought prompting degrades Bengali ( − - 14.4 pp ) and Kannada ( − - 11.4 pp ) rather than helping, exposing English-centric reasoning chains. Aya-Vision-8B, built for 23 languages, still drops 28.5 pp on Dravidian scripts; multilingual pretraining alone does not transfer visual reasoning. I release the translated benchmark and all model outputs. 1 1 1 Code and data: https://github.com/QuantumByte-01/multilingual-vlm-reasoning-audit ; HuggingFace: https://huggingface.co/datasets/Swastikr/multilingual-vlm-reasoning 1 Introduction India’s 260 million school-age children study predominantly in regional-medium schools where the language of instruction is Hindi, Tamil, Telugu, Bengali, Kannada, or Marathi ( Ministry of Human Resource Development, Government of India 2020 ) . EdTech platforms are beginning to integrate vision-language models (VLMs) for automated tutoring and assessment, raising a practical question: can these models reason about mathematics and science in the student’s own language ? Existing benchmarks do not answer this. Indian-language VLM evaluations probe cultural understanding ( Surana et al. 2026 ; Maji et al. 2025 ) , factual VQA ( Faraz et al. 2025 ) , or broad multilingual proficiency ( Khan et al. 2025 ) , but none isolate mathematical, scientific, or spatial reasoning. India-focused VLM architectures such as Chitrarth ( Khan et al. 2025 ) have been proposed, yet no benchmark exists to test whether they, or any other model, can actually solve Kannada science problems or Tamil geometry. I address this gap by translating 980 visual reasoning questions from MathVista ( Lu et al. 2024 ) , ScienceQA ( Lu et al. 2022 ) , and MMMU ( Yue et al. 2024a ) into six Indian languages with IndicTrans2 ( Gala et al. 2023 ) . The images stay fixed across languages; only the question text changes. Eight VLMs are evaluated (GPT-4o, Gemma 3-27B, Llama-4-Maverick, Qwen3-VL-30B, Qwen2.5-VL-32B, Qwen2.5-VL-7B, InternVL2.5-8B, and Aya-Vision-8B), together with text-only and chain-of-thought ablations on Qwen2.5-VL-7B, for a total of 68,600 inference records ( Figure 1 ). Figure 1: Accuracy heatmap across eight models and seven languages. Every model degrades from English (left column) to Indian languages, with Dravidian languages (ta, te, kn) consistently darker than Indo-Aryan (hi, bn, mr). My main findings: • Gemma 3-27B shows the smallest average drop from English (9.8 pp ); GPT-4o achieves the highest Indian-language accuracy (43.2%); open-source models lose 15–25 pp . • Dravidian languages are harder than Indo-Aryan for every model, with the family gap reaching 13.2 pp for Aya-Vision-8B. • Chain-of-thought prompting backfires in Indian languages: Bengali accuracy falls 14.4 pp and Kannada 11.4 pp relative to the standard prompt, indicating that the model’s reasoning chains are locked to English. • Removing the image costs English 15.5 pp but only 4.9–9.7 pp for Indian languages, implying that models under-exploit visual information when text comprehension is weak. • Scaling from 7B to 32B within the Qwen2.5-VL family gains 4.3 pp on Indian languages while losing 1.0 pp on English; scale broadens linguistic coverage without improving reasoning. 2 Related Work Indian-language VLM benchmarks. IndicVisionBench ( Faraz et al. 2025 ) covers 10 Indian languages with over 37K QA pairs but targets cultural understanding and OCR, not reasoning. The BharatBench framework introduced in Khan et al. 2025 spans speech, OCR, and embeddings without isolating STEM tasks. VIRAASAT ( Surana et al. 2026 ) and DRISHTIKON ( Maji et al. 2025 ) evaluate cultural knowledge and geographic diversity. Chitrarth ( Khan et al. 2025 ) introduces an India-focused VLM architecture but no reasoning-specific evaluation. This work differs by holding the image fixed and varying only the question language, so that any accuracy change is directly attributable to language. Multilingual VLM evaluation beyond India. MaRVL ( Liu et al. 2021 ) tests visually grounded reasoning in five languages, including Tamil, but relies on culturally situated images and predates current VLMs. xGQA ( Pfeiffer et al. 2022 ) extends GQA to seven languages, none Indian. VLURes ( Atuhurra et al. 2025 ) benchmarks four languages including Urdu. ALM-Bench ( Vayani et al. 2024 ) and CVQA ( Romero et al. 2024 ) include partial Indian-language coverage but do not focus on reasoning. Pangea ( Yue et al. 2024b ) , a 39-language multimodal LLM, is the closest open-source multilingual effort, though it too lacks a reasoning-focused benchmark. Visual reasoning benchmarks. MathVista ( Lu et al. 2024 ) evaluates mathematical reasoning in visual contexts; MATH-Vision ( Wang et al. 2024 ) extends this to competition-level problems. ScienceQA ( Lu et al. 2022 ) covers K-12 science with diagrams. MMMU ( Yue et al. 2024a ) targets college-level STEM across 30 subjects. All three are English-only; I extend them cross-lingually. Multilingual reasoning in LLMs. Shi et al. 2023 showed that English chain-of-thought outperforms native-language CoT on grade-school math in 10 languages, but without images. Ahuja et al. 2023 broadened multilingual generative evaluation across tasks and languages. Saji et al. 2025 study multilingual reasoning in language reasoning models on text-only benchmarks (MGSM, GPQA), finding that models default to English reasoning chains regardless of prompt language. This paper extends this line of inquiry to the visual domain, where images may partly compensate for weak text comprehension. Indian NLP resources. IndicTrans2 ( Gala et al. 2023 ) provides open-source neural machine translation for all 22 scheduled Indian languages and outperforms Google Translate and NLLB-54B. I use it as the primary translation system, complemented by the broader Indic NLP infrastructure described by Doddapaneni et al. 2023 . 3 Methodology 3.1 Dataset Construction I assemble 980 visual reasoning questions from three sources: • MathVista ( Lu et al. 2024 ) : 400 questions from the testmini split covering geometry, algebra, statistics, chart interpretation, and word problems. Language-independent pattern-matching items (e.g., IQ-test sequences) were excluded. • ScienceQA ( Lu et al. 2022 ) : 400 natural-science questions with images, spanning K-12 physics, chemistry, and biology. • MMMU ( Yue et al. 2024a ) : 180 STEM questions at the college level (mathematics, physics, engineering). Each question is paired with an image that remains identical across languages; only the text changes. 3.2 Languages I select six Indian languages spanning two major families and five scripts ( Table 1 ). Hindi and Marathi share the Devanagari script but differ as languages, enabling us to separate vocabulary and training-data effects from script-level effects. Tamil and Kannada share the Dravidian family but use distinct scripts, providing a complementary control. Table 1: Target languages with script, family, and approximate L1 speaker population. Language Code Script Family L1 (M) English en Latin — — Hindi hi Devanagari Indo-Aryan

Read the complete text