Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modeling

Prahitha Movva, Naga Harshita Marupaka · 2025

Read Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modeling by Prahitha Movva, Naga Harshita Marupaka (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06183.txt.

Introduction

Prahitha Movva and Naga Harshita Marupaka work on answering questions about science figures in a 2025 paper. Reports hold much of their value in charts and figures, not just in text. Models often stumble on exact numbers, multi-step reading of a chart, and keeping text and image claims in line.

The work was built for the SciVQA 2025 shared task. The team tries models with five to eight billion parameters and tests prompt design and step-by-step reasoning. Their best single model scored high on standard text measures. A combined model beat most single models, though the best one still won alone. Error review guides the gains.

Catalog record

CREATORPrahitha Movva, Naga Harshita Marupaka
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06183

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modeling Prahitha Movva, Naga Harshita Marupaka Abstract Technical reports and articles often contain valuable information in the form of semi-structured data like charts, and figures. Interpreting these and using the information from them is essential for downstream tasks such as question answering (QA). Current approaches to visual question answering often struggle with the precision required for scientific data interpretation, particularly in handling numerical values, multi-step reasoning over visual elements, and maintaining consistency between visual observation and textual reasoning. We present our approach to the SciVQA 2025 shared task, focusing on answering visual and non-visual questions grounded in scientific figures from scholarly articles. We conducted a series of experiments using models with 5B to 8B parameters. Our strongest individual model, InternVL3, achieved ROUGE-1 and ROUGE-L F1 scores of 0.740 and a BERTScore of 0.983 on the SciVQA test split. We also developed an ensemble model with multiple vision language models (VLMs). Through error analysis on the validation split, our ensemble approach improved performance compared to most individual models, though InternVL3 remained the strongest standalone performer. Our findings underscore the effectiveness of prompt optimization, chain-of-thought reasoning and ensemble modeling in improving the model’s ability in visual question answering. 1 Introduction Scientific literature communicates complex ideas not only through text but also through carefully designed visual elements including charts, graphs, diagrams, and technical illustrations. These visualizations serve as dense information carriers, encoding quantitative relationships, experimental results, architectural designs, and conceptual frameworks that are essential for scientific understanding. The ability to automatically interpret and reason about these visual elements represents a critical challenge in advancing scientific AI systems. The task of Visual Question Answering (VQA) over scientific figures presents unique challenges that distinguish it from general-domain VQA. Scientific visualizations demand mathematical precision, often requiring exact numerical extraction and calculation. They involve complex compositional reasoning across multiple visual elements, and frequently contain domain-specific conventions, symbols, and representations that require specialized understanding Ishmam et al. 2024 . Furthermore, scientific figures often embed multiple layers of information, including raw data points, derived trends, statistical relationships, and comparative analyses. Current VQA models, while showing impressive performance on general datasets, often struggle with the precision and reasoning depth required for scientific applications Kabir et al. 2024 . Common failure modes include visual grounding errors, where models misinterpret chart elements or scales; compositional reasoning failures, where multi-step logical processes break down; and consistency issues between visual observations and textual explanations Tanjim et al. 2025 ; Thawakar et al. 2025 . This paper presents our approach to the SciVQA Shared Task 1 1 1 https://sdproc.org/2025/scivqa.html 2 2 2 https://huggingface.co/datasets/katebor/SciVQA Borisova et al. 2025 , focusing on QA over scientific visualizations. The task involves answering closed-ended visual (i.e., addressing visual attributes such as colour, shape, size, height, etc.) and non-visual (not addressing figure visual attributes) questions. We leverage the reasoning and visual understanding capabilities of VLMs, and employ task-specific Chain-of-Thought (CoT) Wei et al. 2023 prompting techniques to retrieve and summarize relevant information from the visualizations. Our approach involved testing multiple prompt variants and selecting optimal configurations based on validation performance. To support reproducibility and future research, we make the code publicly available on GitHub 3 3 3 https://github.com/NagaHarshita/Infyn-SciVQA . Our main contributions are: • A systematic ensemble strategy with figure type specific model selection based on comprehensive validation analysis. • Optimized prompt engineering templates tailored to different question answer pair and figure type combinations. 2 Related Work Recent advancements in chart-based QA have focused on various approaches to understanding and generating responses about visualizations. ChartLlama Han et al. 2023 and UniChart Masry et al. 2023 demonstrate the benefits of chart-specialized language models, showing improved performance in both chart captioning and QA tasks. These works often rely on explicit chart structure parsing as a preprocessing step, achieving strong results on synthetic chart datasets. Chart-based Reasoning Carbune et al. 2024 and LlamaV-o1 Thawakar et al. 2025 propose decomposed reasoning traces and transfer of LLM capabilities to visual settings. Our work builds on these insights by emphasizing structured reasoning and adopt step-level supervision to encourage coherent and faithful intermediate reasoning in visual contexts. Other relevant works include SPIQA Pramanick et al. 2025 and MathVista Lu et al. 2024 , which evaluate visual reasoning in scientific domains. MathVista particularly focuses on precise numerical and symbolic interpretation in mathematical visualizations, similar to our emphasis on scientific accuracy. In the realm of prompt engineering and model alignment, Zhan et al. 2025 proposed SPRI (Situated-PRInciples), a framework that automatically generates context-specific guiding principles for each input query to improve model alignment. Their approach demonstrates that instance-specific principles can outperform generic ones, which informs our ensemble methodology that combines prompt engineering with multiple VLMs. Motivated by recent advances in prompt rewriting Tanjim et al. 2025 , we explore instruction tuning and prompt optimization to enhance model adherence to scientific QA formats. Unlike previous work that focuses on architectural innovations requiring additional training, our approach focuses on ensemble strategies and prompt optimization for maximum performance on scientific VQA tasks. 3 Dataset The SciVQA dataset comprises scientific figures from ACL Anthology and arXiv papers. Each figure is annotated with seven question-answer pairs and associated metadata including captions, figure IDs, figure types (e.g., compound, line graph, bar chart, scatter plot), and QA pair types, with dataset splits and distributions detailed in Section A and Tables 3 , 4 , and 5 . 4 Methodology Our system integrates three key components: • systematic prompt optimization for different figure types • strategic ensemble modeling, and • post-processing for answer standardization. We utilized the vLLM Kwon et al. 2023 engine for maximum compute utilization during inference. A40 instances were sufficient for 7B models, while 8B models required A100 GPUs. CoT inference required approximately twice the computation time due to the two-level reasoning process, but provided significant quality improvements. 4.1 Model Selection To inform model selection and ensure alignment with the target domain, we referred to the performance of recent models on established multimodal QA benchmarks analogous to SciVQA Borisova et al. 2025 , including ChartQA Masry et al. 2022 , MathVista Lu et al. 2024 , ChartXiv Wang et al. 2024 . VLMs in the 5–8B parameter range demonstrated competitive performance on these leaderboards, achieving results comparable to significantly larger models with 32–72B parameters. According to the InternVL3 technical report Zhu et al. 2025 , the models InternVL3-8B and Qwen2.5-VL-7B performed well on tasks such as OCR, chart, and document understanding, specifically on datasets like ChartXiv and ChartQA. Additionally, a fine-tuned version of the Qwen2.5-VL-7B Instruct model Bai et al. 2025 , as reported in the Bespoke technical report 4 4 4 https://www.bespokelabs.ai/blog/bespoke-minichart-7b , demonstrates competitive performance on ChartXiv, ChartQA, and EvoChart, achieving results comparable to InternVL3-8B. Based on these observations, we chose the following four VLMs, namely, InternVL3-8B, Qwen2.5-VL-7B Instruct, Bespoke MiniChart 7B, and Phi-4 Multimodal Instruct for our task. InternVL3-8B Zhu et al. 2025 features an advanced vision encoder architecture tailored for complex visual understanding, unlike Qwen2.5-VL which uses a standard ViT encoder Zhu et al. 2025 . The model supports high-resolution image processing capabilities essential for interpreting detailed charts, and incorporates multi-scale feature extraction to enable both global comprehension and fine-grained numerical

Read the complete text