AGIC: Attention-Guided Image Captioning to Improve Caption Relevance

L. D. M. S. Sai Teja Ashok Urlana Pruthwik Mishra · 2025

Read AGIC: Attention-Guided Image Captioning to Improve Caption Relevance by L. D. M. S. Sai Teja Ashok Urlana Pruthwik Mishra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06853.txt.

Introduction

Sai Teja and co-authors improve image captioning by steering the model toward the most telling parts of a picture, in a 2025 paper. Automatic captions often miss the main subject or add vague filler. Relevance, not just fluency, is the weak spot. Their method, AGIC, strengthens salient regions right inside the feature space.

A hybrid decoding scheme mixes steady choices with sampled variety. That balance keeps captions natural yet diverse. Experiments on two standard Flickr data sets compare AGIC with leading models. It matches or beats them on several metrics while running faster. The design stays interpretable, since attention maps show what drove each word. Accessibility tools for blind users stand to gain from better captions.

Catalog record

CREATORL. D. M. S. Sai Teja Ashok Urlana Pruthwik Mishra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06853

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AGIC: Attention-Guided Image Captioning to Improve Caption Relevance L. D. M. S. Sai Teja Ashok Urlana Pruthwik Mishra Abstract Despite significant progress in image captioning, generating accurate and descriptive captions remains a long-standing challenge. In this study, we propose Attention-Guided Image Captioning (AGIC), which amplifies salient visual regions directly in the feature space to guide caption generation. We further introduce a hybrid decoding strategy that combines deterministic and probabilistic sampling to balance fluency and diversity. To evaluate AGIC, we conduct extensive experiments on the Flickr8k and Flickr30k datasets. The results 1 1 1 https://github.com/saitejalekkala33/AGIC-code show that AGIC matches or surpasses several state-of-the-art models while achieving faster inference. Moreover, AGIC demonstrates strong performance across multiple evaluation metrics, offering a scalable and interpretable solution for image captioning. 1 Introduction Image captioning, a prominent task in computer vision, aims to generate a visually grounded description of an image Cornia et al. 2020 ; Cornia et al. 2019 ; Chen et al. 2015 . While significant performance improvements have been achieved, current methods frequently generate generic captions, limiting their utility in capturing nuanced visual details Al Badarneh et al. 2025 ; Ma et al. 2023 ; Guo et al. 2020 ; Lu et al. 2018 . The core challenge in producing relevant and descriptive image captions stems from the inherent difficulty in comprehensively capturing every visual aspect within an image. LlaVA: Two little girls are playing with bubbles in the park. Qwen: Two young friends share a joyful moment, creating a world of bubbles in the park. Fuyu: Two little girls blowing bubbles in the park. BRNN: Two girls playing in the park. LSTNet: A young girl is blowing bubbles, holding them in her hands. \bm{R}^{2}\bm{M} : Two young girls playing with bubbles in grass. AGIC : Two young girls wearing floral dress blowing bubbles in a park covered with grass. Figure 1: Comparison of various image caption generation models. red: zero-shot, cyan: supervised, violet: unsupervised approaches and blue: our approach. To address limitations in caption relevancy and descriptiveness, existing approaches are broadly categorized into three main types: supervised, unsupervised, and semi-supervised methods. Supervised methods rely on large-scale, manually annotated image-caption pairs for training Xu et al. 2015 ; Cornia et al. 2019 ; Anderson et al. 2018 . While they often yield highly accurate captions, they are resource-intensive, requiring significant time and costly “gold-standard” data. In contrast, unsupervised methods utilize unpaired image sets and independent text corpora to learn visual concept detection and cross-modal alignment without explicit image-caption pairings Feng et al. 2019 ; Guo et al. 2020 ; Laina et al. 2019 . However, these models typically generate less precise or coherent captions due to the absence of direct supervision. Finally, semi-supervised methods Liang et al. 2024 attempt to bridge this gap by generating pseudo-labels for unlabeled data, which are then used to guide the learning process. Although less data-hungry, these methods can suffer from error propagation if the pseudo-labels are inaccurate. To overcome these challenges, in this study, we propose a novel attention-guided training-free approach by amplifying the pretrained model’s attention weights to capture the relevant regions of the image along with a hybrid decoding approach. The proposed approach offers a compelling alternative by significantly reducing reliance on costly paired datasets, making them particularly attractive for scenarios where extensive ground-truth data is scarce. As shown in Figure 1 , the zero-shot and unsupervised-based approaches tend to generate generic captions, whereas the supervised methods are unable to capture most of the relevant objects present in the image. In contrast, the proposed AGIC approach covers all the relevant objects present in the image and provides grammatically fluent and descriptive caption. Our key contributions are: 1) A novel attention-guided image caption approach designed to generate relevant and descriptive captions. 2) A novel hybrid decoding strategy for enhanced caption generation. 3) Extensive experiments conducted on two popular image caption datasets using several Vision-Language Models (VLMs), along with rigorous ablation studies. 2 Related Work Image captioning is a challenging task, and several efforts have been made to solve this problem. A family of attention-based approaches Anderson et al. 2018 ; Huang et al. 2019 ; Cornia et al. 2020 ; Sharma et al. 2018 have been incorporated for the object detection by capturing attributes, and other characteristics in images. Visual attention-based approaches focus on salient regions in an image using an object relation transformer that captures spatial dependencies between the detected objects Herdade et al. 2019 . In the same line, Anderson et al. 2018 propose bottom-up and top-down attention to combine object-level visual features with textual words. Huang et al. 2019 extends this using Attention-on-Attention (AoA) with an additional gating mechanism to determine the relevance of the final attention results and the incoming queries. More recently, to infuse relevancy in image captioning, Honda et al. 2021 tackle spurious word-level alignments in pseudo-captioning by implementing a gating mechanism to filter out irrelevant visual words. Shi et al. 2021 develops a descriptive image captioning model using Natural Language Inference (NLI) between all pairs of reference captions. To better capture semantic information, Shi et al. 2020 constructs visual relationship graphs guided by captions. They also develop a multitask and weakly multi-instance learning framework for accurate explicit object and predicate detection. Another notable contribution by Pan et al. 2020 involves the modification of the conventional attention block into X-Linear attention networks to learn second-order feature interactions using bilinear pooling. Although supervised approaches have achieved state-of-the-art performance, they rely on large paired datasets, whereas unsupervised efforts Feng et al. 2019 ; Gu et al. 2019 ; Honda et al. 2021 suffer from semantic drift and coherence. In contrast, we propose a training-free approach by amplifying the pretrained model’s attention weights to capture the relevant regions of the image and generate descriptive captions. 3 Model Description We propose a framework to improve image caption relevance using a contextual relevance amplification mechanism, implemented through an attention-guided process. Our approach is inspired by Liu et al. 2025 , who used attention patterns and self-reflection to detect hallucinations in large language models without relying on labeled data. Dataset Approach B1 B2 B3 B4 R-L METEOR CIDEr SPICE Flickr8k BLIP2-opt-2.7b Li et al. 2023 0.391 0.255 0.163 0.102 0.325 0.259 0.077 0.041 LLaVA-1.5B-7B Liu et al. 2023 0.440 0.293 0.193 0.128 0.357 0.049 0.198 0.080 Qwen2.5-VL-7B-Instruct Bai et al. 2025 0.441 0.276 0.170 0.107 0.311 0.034 0.280 0.072 Fuyu-8B Bavishi et al. 2023 0.630 0.448 0.302 0.201 0.147 0.441 0.414 0.12 BRNN Karpathy and Fei-Fei 2015 0.563 0.362 0.219 0.131 0.350 0.171 0.512 0.112 R^{2}M Guo et al. 2020 0.496 0.302 0.177 0.081 0.320 0.132 0.284 0.030 LSTNet Ma et al. 2023 0.669 0.448 0.304 0.241 0.417 0.210 0.623 0.137 Ensemble Al Badarneh et al. 2025 0.728 0.495 0.323 0.208 0.432 0.235 0.604 0.164 Ours (AGIC + LLaVA-1.5B-7B) 0.651 0.478 0.342 0.239 0.442 0.242 0.697 0.213 Ours ( AGIC + BLIP2-opt-2.7b ) 0.665 0.499 0.355 0.251 0.248 0.445 0.734 0.195 Flickr30k BLIP2-opt-2.7b Li et al. 2023 0.406 0.260 0.164 0.103 0.306 0.245 0.083 0.083 LLaVA-1.5B-7B Liu et al. 2023 0.471 0.316 0.213 0.145 0.348 0.070 0.234 0.096 Qwen2.5-VL-7B-Instruct Bai et al. 2025 0.454 0.277 0.166 0.100 0.285 0.055 0.254 0.088 Fuyu-8B Bavishi et al. 2023

Read the complete text