SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering

Yan Zhang, Jiaqing Lin, Miao Zhang, Kui Xiao, Xiaoju Hou, Yue Zhao, Zhifei Li (✉) · 2025

Read SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering by Yan Zhang, Jiaqing Lin, Miao Zhang, Kui Xiao, Xiaoju Hou, Yue Zhao, Zhifei Li (✉) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20871.txt.

Introduction

Answering questions about an image often needs outside knowledge, and image captions are a noisy bridge to that knowledge. Yan Zhang and six co-authors present SCRA-VQA in a 2025 paper accepted at the DASFAA 2025 conference. The method improves knowledge-based visual question answering.

A vision-language model first turns the image into a caption. SCRA-VQA then summarizes the caption and reranks its parts, keeping what the question actually needs. It also generates contextual examples to guide the language model's reasoning. This cleaning step counters the excess noise that raw captions carry. The authors report stronger answers from the augmented language model. The approach shows how careful text preparation can beat simply feeding a model more text.

Catalog record

CREATORYan Zhang, Jiaqing Lin, Miao Zhang, Kui Xiao, Xiaoju Hou, Yue Zhao, Zhifei Li (✉)
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20871

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering Yan Zhang, Jiaqing Lin, Miao Zhang, Kui Xiao, Xiaoju Hou, Yue Zhao, Zhifei Li (✉) Abstract Acquiring high-quality knowledge is a central focus in Knowledge-Based Visual Question Answering (KB-VQA). Recent methods use large language models (LLMs) as knowledge engines for answering. These methods generally employ image captions as visual text descriptions to assist LLMs in interpreting images. However, the captions frequently include excessive noise irrelevant to the question, and LLMs generally do not comprehend VQA tasks, limiting their reasoning capabilities. To address this issue, we propose the Summarized Caption-Rerank Augmented VQA (SCRA-VQA), which employs a pre-trained visual language model to convert images into captions. Moreover, SCRA-VQA generates contextual examples for the captions while simultaneously summarizing and reordering them to exclude unrelated information. The caption-rerank process enables LLMs to understand the image information and questions better, thus enhancing the model’s reasoning ability and task adaptability without expensive end-to-end training. Based on an LLM with 6.7B parameters, SCRA-VQA performs excellently on two challenging knowledge-based VQA datasets: OK-VQA and A-OKVQA, achieving accuracies of 38.8% and 34.6%. Our code is available at https://github.com/HubuKG/SCRA-VQA . 1 Introduction Visual Question Answering (VQA) is a crucial task that combines natural language processing and computer vision. Over the past few years, substantial progress has been made in the field of VQA [ 1 , 2 ] . Among the VQA tasks, knowledge-based visual question answering (KB-VQA) is particularly challenging, requiring external knowledge beyond the given question and visual content to provide accurate answers. To solve such a task, a model must possess strong visual perception and reasoning abilities, along with the capacity to effectively integrate world knowledge from external knowledge bases (e.g., Wikipedia). Therefore, accurately acquiring the required knowledge has become the core problem of KB-VQA. In the current study, researchers focus on image recognition and knowledge-based logical reasoning [ 3 ] . Image recognition aims at obtaining more visual information to help understand user questions [ 4 ] . Logical reasoning aims to identify associations between knowledge to reason the answer accurately [ 6 ] . Simultaneously, large language models (LLMs) like GPT-3 have demonstrated excellent capabilities in knowledge retrieval and question-answering tasks in natural language processing. Inspired by these LLMs, Prophet [ 7 ] achieves better results on knowledge-based VQA tasks by prompting GPT-3 for reasoning, outperforming traditional large vocabulary models. Repurposing pre-trained LLMs for new tasks is ideal from a resource perspective because it avoids the high computational costs of traditional model-building methods. Figure 1 : Comparison between existing prompting methods and SCRA-VQA on VQA tasks using frozen LLMs. Unlike previous methods that provide image-to-text information directly into LLMs, SCRA-VQA processes information in three distinct ways before passing it on to help better understand the task. However, as the demand for solving more complex problems grows, the knowledge required for knowledge-based logical reasoning becomes increasingly complex. Meanwhile, the mismatch between the generated image description and the question may make the LLMs less powerful. For instance, a caption that says, “ a group of puppies playing on the grass ” would be unhelpful for a question like “ What season is depicted in the image? ” and might even hinder the LLMs from generating the correct answer. In addition, since LLMs only take in text, the precision of their logical reasoning is almost dependent on the acquired information. Relying only on the captions of a few relevant entities may lead to incorrect judgment and reduced performance. Thus, activating the performance of LLMs is a crucial issue. To address these issues, this paper proposes a Summarized Caption-Rerank Augmented VQA (named SCRA-VQA) method based on frozen LLMs. As shown in Fig. 1 , it demonstrates the VQA task and conveys the image’s content to the LLM for answering questions. Compared to previous methods, our method uses a pre-trained vision-language model [ 8 ] to generate captions for images. Then, a question generation model is used to convert the captions into similar answer-question pairs as contextual examples [ 9 ] , enabling the LLM to understand the VQA task better. Furthermore, considering the large number of generated captions and the lack of connection between them, we summarized and Rerank it to enable LLMs to understand image information while reducing data processing stress fully. In this paper, we have made the following contributions: • We propose a Summarized Caption-Rank Augmented VQA (SCRA-VQA) method, a new method for processing captions to assist LLM in VQA tasks. This method ensures that captions are only relevant to the information needed for the question, enabling better connections between vision and language modalities. • SCRA-VQA enables pre-trained LLMs to execute zero-shot VQA without requiring costly end-to-end training or specialized text networks. This method facilitates low-cost and adaptable model deployment, providing an effective and lightweight solution. • Our experimental results demonstrate that the OPT model effectively achiev-es zero-shot VQA when equipped with SCRA-VQA. Moreover, our method outperforms the state-of-the-art Img2Prompt methodology on the A-OKVQA dataset by 3.9%. 2 Related Work 2.1 Visual Question Answering VQA has always been a central focus of research. Recent VQA work mainly focuses on two areas: 1) better visual feature extraction and 2) more advanced model architectures. Visual feature extraction has always been a key research direction in VQA. Numerous methods continuously explore how to more effectively utilize visual information to improve the performance of question-answering systems. For instance, the bottom-up and top-down attention mechanism [ 12 ] allows attention to be calculated at the level of visual objects. CLIP-ViL [ 13 ] introduces a novel training methodology using a contrastive loss function to learn visual concepts directly from natural language descriptions. This enables zero-shot solid performance across a broad range of visual tasks without requiring task-specific training. VinVL [ 14 ] improves visual representations in vision-language tasks by developing an enhanced object detection model pre-trained on extensive datasets. Overall, these innovative methods collectively enhance the effectiveness and robustness of visual feature extraction in VQA tasks. Recent VQA frameworks have made substantial progress by integrating visual and linguistic information. Among these, BLIP [ 8 ] introduces a vision-language pre-training framework that enhances performance by generating synthetic captions and filtering noise from web-derived data. N2NMNs [ 15 ] develop End-to-End Module Networks that autonomously generate dynamic layouts from textual queries for efficient visual question answering. Additionally, certain studies have concentrated on neurosymbolic VQA, which involves deriving a structural scene representation from an image and generating a program sequence based on the question. In particular, VQs [ 16 ] gather visual questions and segmentation answers, establishing a connection between instance segmentation and the question-answer pairs in the VQA dataset. NS-VQA [ 17 ] introduces a model that identifies scene objects and employs a symbolic program executor for reasoning and responding to inquiries. DCR [ 18 ] suggests a profound concept reasoner to create syntactic rule structures from concepts and apply these rules to meaningful concepts for interpretable predictions. These innovative methods collectively enhance the accuracy and interpretability of the visual-to-text process in VQA tasks. Based on these methods, our method demonstrates that image-to-text transforms and the critical information of the question can complement each other. This synergy provides more complete visual

Read the complete text