Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model Journal: Journal

Reem AlJunaid, Muzammil Behzad · 2025

Read Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model Journal: Journal by Reem AlJunaid, Muzammil Behzad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23358.txt.

Introduction

Reem AlJunaid and Muzammil Behzad present a model that writes rich, knowledge-filled captions for images. Their paper appeared in 2025. Many caption systems produce bland lines that miss context. Their model replays stored knowledge while it writes, and uses beam search to draft several candidate captions before choosing. Attention modules inside the image encoder sharpen which features the writer sees.

Training schedules smooth the learning process. The authors report gains in both caption quality and recognition of the facts an image shows. The model also handled knowledge ideas it had not seen in training. They present the design as a step toward captions that inform, rather than merely label, useful for search, teaching, and access tools for blind users.

Catalog record

CREATORReem AlJunaid, Muzammil Behzad
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23358

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model Journal: Journal Reem AlJunaid, Muzammil Behzad Abstract Generating informative and knowledge-rich image captions remains a challenge for many existing captioning models, which often produce generic descriptions that lack specificity and contextual depth. To address this limitation, we propose KRCapVLM, a knowledge replay-based novel image captioning framework using vision-language model. We incorporate beam search decoding to generate more diverse and coherent captions. We also integrate attention-based modules into the image encoder to enhance feature representation. Finally, we employ training schedulers to improve stability and ensure smoother convergence during training. These proposals accelerate substantial gains in both caption quality and knowledge recognition. Our proposed model demonstrates clear improvements in both the accuracy of knowledge recognition and the overall quality of generated captions. It shows a stronger ability to generalize to previously unseen knowledge concepts, producing more informative and contextually relevant descriptions. These results indicate the effectiveness of our approach in enhancing the model’s capacity to generate meaningful, knowledge-grounded captions across a range of scenarios. 1 Introduction Image captioning is the task of generating descriptions of images using computer vision and natural language processing techniques [ 1 , 2 , 3 ] . It has a variety of applications, such as enhancing the content understanding of multimedia [ 4 , 5 ] and helping visually impaired people [ 6 , 7 , 8 , 9 , 10 , 11 ] . Current existing models, however, often produce generic captions which lack real-world concepts like contextual details and named entities [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] . This missing knowledge often represents key information in understanding the content of the image. Furthermore, this detailed knowledge can also enhance the performance of other models that rely on the output of image captioning systems, such as question answering systems [ 22 , 23 , 24 , 25 , 26 ] . Several efforts have been placed in order to enhance the descriptions with real-world knowledge [ 13 , 27 , 28 ] . However, most of these works are limited by using external resources, such as image metadata or object recognition models, to detect existing entities before generating the descriptions. To overcome that, the Vision-Language Pretrained (VLM) models [ 29 , 30 ] present a powerful solution for this era. They were trained on massive data and can capture diverse real-world knowledge [ 29 , 31 , 30 , 32 , 33 , 34 ] . Despite their tremendous performance, VLM models suffer from two main problems: (1) zero-shot inference leading to safe but low-quality descriptions [ 35 , 36 , 37 ] , and (2) knowledge hallucination due to the noise in image-text pairs in pre-training [ 35 , 36 ] . Additionally, fine-tuning VLM models on downstream tasks introduces a “generic bias” that restricts their expression of detailed knowledge [ 37 ] . Addressing these limitations, the Knowledge guided Replay (K-Replay) framework is proposed in [ 38 ] . K-Replay preserves the original model structure while helping VLM models retain knowledge during fine-tuning on downstream tasks. It is done by selecting knowledge-rich samples from the pretraining data and computing a coverage loss based on the existence of required keywords in the sentence to reinforce memory of this knowledge, and hence, avoid generic descriptions. Moreover, to reduce hallucination and ensure faithful descriptions, a knowledge distillation-based constraint is applied from a fine-tuned model. With strong performance, especially in unseen scenarios, K-Replay effectively helps the model to recall and express pre-learned knowledge. Despite these strengths, K-Replay has several limitations. First, it generates pseudo-captions for replay samples using greedy decoding, which tends to produce low-diversity and overly generic sentences. These captions may fail to capture detailed knowledge, thereby weakening the replay mechanism’s effectiveness. Second, the combination of multiple loss terms in its framework often leads to unstable training dynamics and convergence issues. Finally, K-Replay lacks explicit self-attention among image patches, limiting its ability to capture fine-grained visual patterns essential for real-world knowledge. Furthermore, this work focuses on enhancing VLM models to produce knowledge-rich and accurate image captions by evaluating using the KnowCap [ 38 ] dataset. As such, the scope of knowledge assessed is limited to the types of real-world knowledge represented by the predefined keyword categories in the KnowCap dataset. These include four categories, foods, brands, landmarks, and movie characters. The study does not attempt to capture or evaluate all forms of general knowledge but it focuses on the model’s ability to incorporate specific knowledge keywords, as defined by this benchmark, into the generated captions. 2 Literature Review Image captioning has evolved through various techniques, starting from traditional encoder-decoder architecture to transformer-based architecture [ 39 , 40 , 41 , 42 ] . In traditional techniques, images are encoded with convolutional neural networks (CNNs) and texts decoded with recurrent neural networks (RNNs), converting visual features into language linearly. Even though these models are effective, they often fail to capture extended dependencies within captions and the complexity of visual scenes. On the other hand, transformer-based frameworks offer attention mechanisms for identifying semantic relations within visual scenes rather than considering them separately. Using self-attention, transformers create accurate, knowledge-rich captions that capture detailed scenes despite lengthy or complex descriptions. Compared to traditional approaches, these transformer-based models provide superior accuracy and knowledge-rich captions. Traditional encoder-decoder architecture plays a key role in the development of image captioning models. For instance, Vinyals et al. [ 3 ] pioneered the use of a deep learning encoder-decoder architecture for image captioning. Their approach involved using pre-trained CNNs for image encoding, with the last hidden layer output passed into an RNN to generate descriptive captions. Their work provided a baseline for subsequent studies, which have expanded and refined the model by incorporating various enhancements, including the visual attention mechanism introduced by [ 43 ] . Another popular framework that serves as a baseline for many image captioning models is the Bottom-Up and Top-Down framework [ 1 ] . By leveraging Faster R-CNN, it combines two types of attention: Bottom-Up, which identifies important parts of the image using feature vectors, while Top-Down decides which parts to focus on while generating the next word of the caption. Similarly, Huang et al. [ 44 ] developed the attention-on-attention (AoA) framework, which improves traditional attention by adding another layer to help the model choose more relevant words based on context. Another encoder-decoder-based framework is the Deep Hierarchical Encoder-Decoder Network (DHEDN) [ 45 ] . DHEDN comprises a three-layer LSTM architecture: S-LSTM handles text encoding, VSE-LSTM combines visual and textual features into a common semantic space, and SF-LSTM is responsible for generating captions. Most recently, research in image captioning has shifted towards transformer-based architecture. The self-attention in deep learning models enables simultaneous processing of images and captions, thus helping generate more accurate and knowledge-rich descriptions [ 40 ] . Unlike RNNs, transformers process sequences in parallel, making training much faster and more scalable. In this regard, Vaswani et al. [ 46 ] introduced self-attention for efficient parallel processing, replacing recurrence entirely. The model treats image features as input tokens and uses standard encoder-decoder layers for generation.

Read the complete text