From Image Captioning to Visual Storytelling

Admitos Passadakis, Yingjin Song, Albert Gatt · 2025

Read From Image Captioning to Visual Storytelling by Admitos Passadakis, Yingjin Song, Albert Gatt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14045.txt.

Introduction

Stories for image sequences can be built by captioning first, Admitos Passadakis, Yingjin Song, and Albert Gatt show in a 2025 paper. Visual storytelling asks a system to generate a story for a stream of images. The story should be grounded in the sequence yet also narrative and coherent, a balance that makes the task hard for direct end-to-end generation.

Their approach treats storytelling as a superset of image captioning. A vision-to-language model first captions the input images. Language-to-language methods then transform those captions into coherent narratives. A multifarious evaluation shows the unified framework improves story quality. Training time falls, and the framework is readily reusable. A new metric called ideality measures distance from an oracle model.

Catalog record

CREATORAdmitos Passadakis, Yingjin Song, Albert Gatt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-14045

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Image Captioning to Visual Storytelling Admitos Passadakis, Yingjin Song, Albert Gatt Abstract Visual Storytelling is a challenging multimodal task between Vision & Language, where the purpose is to generate a story for a stream of images. Its difficulty lies on the fact that the story should be both grounded to the image sequence but also narrative and coherent. The aim of this work is to balance between these aspects, by treating Visual Storytelling as a superset of Image Captioning, an approach quite different compared to most of prior relevant studies. This means that we firstly employ a vision-to-language model for obtaining captions of the input images, and then, these captions are transformed into coherent narratives using language-to-language methods. Our multifarious evaluation shows that integrating captioning and storytelling under a unified framework, has a positive impact on the quality of the produced stories. In addition, compared to numerous previous studies, this approach accelerates training time and makes our framework readily reusable and reproducible by anyone interested. Lastly, we propose a new metric/tool, named ideality , that can be used to simulate how far some results are from an oracle model, and we apply it to emulate human-likeness in visual storytelling. 1 Introduction The task of generating a textual story given a series of input images that are potentially correlated, is called Visual Storytelling Huang et al. 2016 . This is a challenging multimodal task that lies in the intersection of Computer Vision (CV) and Natural Language Processing (NLP). In most cases, Visual Storytelling is regarded to be a distinct task from Image Captioning Lin et al. 2014 , which focuses on describing objects and their attributes. Indeed, Visual Storytelling is not only about mere image descriptions, as it aims to bridge the semantic gap between visual data and textual representations by introducing a narrative form, coherence and a sense of chronological progress in the output text. While Image Captioning models have performed well in generating detailed descriptions of individual images Anderson et al. 2018 ; Lu et al. 2019 , storytelling necessitates more sensitivity to temporal relationships and more narrative structure. Existing visual storytelling methods have mainly focused on end-to-end systems, which map images sequences directly into stories Wang et al. 2018a ; Smilevski et al. 2018 ; Kim et al. 2019 ; Su et al. 2020 ; Jung et al. 2020 ; Yu et al. 2021 , by utilizing typical visual-encoder & text-decoder architectures. While these approaches have been promising regarding visual grounding , they often fall short in resolving two other crucial components of storytelling: narrative coherence and semantic richness Liu et al. 2017 ; Park and Kim 2015 . The issue of joint optimization of these dimensions has given rise to various solutions, from knowledge-enriched techniques Hsu et al. 2019a ; Hsu et al. 2021 ; Chen et al. 2022a to more recent applications of multi-modal Large Language Models (LLMs) Liu et al. 2023b ; Zhang et al. 2024 . Here, we propose an alternative to this challenge by addressing Visual Storytelling formally as a superset of Image Captioning. Our solution splits storytelling into two separate stages: first, we create precise and expressive descriptions of single images with an advanced captioning model, and then, we convert these descriptions into coherent stories using sophisticated language-to-language methods. This decomposition has several advantages: it leverages the maturity of image captioning models, simplifies storytelling by separating visual understanding from narrative generation, and makes the overall system more reproducible 1 1 1 In fact, our system can be tested online by clicking HERE . Please, notify the first author before proceeding to usage. . Another challenge in visual storytelling is that there is no objectively correct and acceptable way of measuring which story can be considered as complete. Despite that visual grounding is a commonly admissible dimension, the same cannot be said for narrative coherence and semantic richness . Different studies have attempted to capture all these metrics, by recruiting different methods, algorithms or language models Wang et al. 2022 ; Surikuchi et al. 2023 . To overcome this hurdle, we additionally propose a novel metric that can be utilized in conjunction with other metrics, that measure similar characteristics, and with one oracle method, such as human writing, that is objectively accepted as a comprehensive way of written text. As a result, we believe that this new metric could simulate some aspects of human-likeness in the generated stories. Having said these, we can summarize the main contributions of this research as follows: • We propose a simple, yet very effective visual storytelling method which is lightweight, easily reusable and reproducible. Its efficiency is shown through multiple evaluation methods. • We successfully unify two different Vision & Language tasks, Image Captioning and Visual Storytelling, under a single-stream pipeline. • We also propose a new metric/tool that can be universally used in order to measure how close we are to an ideal result. In our case, we utilize it to simulate how human-like are our generated stories in linguistic terms. 2 Related Work 2.1 Image Captioning Image captioning has evolved from early hand-tuned approaches based on manually crafted features and linguistic rules Farhadi et al. 2010 ; Devlin et al. 2015 to modern deep learning methods like CNNs for feature extraction and RNNs for sequential text generation. The introduction of Microsoft COCO dataset Lin et al. 2014 , provided large-scale, diverse image-caption pairs, allowing standard evaluation metrics to be set and models to be benchmarked Vinyals et al. 2015 . Subsequent advancements introduced attention mechanisms which enabled models to focus on the appropriate regions of an image Xu et al. 2016 ; Lu et al. 2017 . Reinforcement Learning (RL) for optimizing captioning models was added in subsequent enhancements Rennie et al. 2017 , whilst graph-based approaches utilized scene graphs to facilitate contextual understanding Yang et al. 2018 ; Yao et al. 2018 . The transition to transformers-based methods also introduced new improvements, such as the attention-on-attention networks Huang et al. 2019 and entangled spatial-aware Transformers Herdade et al. 2020 ; He et al. 2020 . 2.2 Visual Storytelling Visual Storytelling (VIST) Huang et al. 2016 is a challenging multimodal task that aims to generate coherent and human-like narratives from sequences of ordered images and it is often differentiated from Image Captioning. Initial attempts built upon image captioning models, incorporating global-local visual attention Kim et al. 2019 and contextualized image representations Gonzalez-Rico and Fuentes-Pineda 2018 to improve narrative fluency. Traditional methods relied on CNNs and LSTMs Kim 2015 ; Yu et al. 2017 ; Yu et al. 2021 , while more recent approaches leverage Transformer architectures to integrate image and text features Qi et al. 2021 ; Fan et al. 2022 . RL was also used for VIST, with custom reward functions guiding story generation Wang et al. 2018a ; Wang et al. 2018b ; Huang et al. 2019 ; Hu et al. 2020 ; Chen et al. 2024 , though its training instability remains a challenge. Beyond data-driven models, external knowledge integration played a crucial role in enhancing story richness and logical coherence, enabling models to reason about commonsense concepts that are not explicitly present in images Yang et al. 2019 ; Hsu

Read the complete text