The Devil is in the EOS: Sequence Training for Detailed Image Captioning
Abdelrahman Mohamed, Yova Kementchedjhieva · 2025
Read The Devil is in the EOS: Sequence Training for Detailed Image Captioning by Abdelrahman Mohamed, Yova Kementchedjhieva (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20077.txt.
Introduction
Abdelrahman Mohamed and Yova Kementchedjhieva trace a 2025 flaw in image caption tools: they stop too soon. Even strong vision models write short, plain captions, though they could say far more. The team pins the cause on training, which teaches the model to favor the end-of-text token too early.
Their fix needs no new data and no reward design. By simply damping that early-stop bias in a trained model, captions grew longer and richer across three models and three test sets. The trade-off was more made-up detail, as expected. Any lab can apply the trick to its own model at once.
Catalog record
| CREATOR | Abdelrahman Mohamed, Yova Kementchedjhieva |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-20077
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Devil is in the EOS: Sequence Training for Detailed Image Captioning Abdelrahman Mohamed, Yova Kementchedjhieva Abstract Despite significant advances in vision-language models (VLMs), image captioning often suffers from a lack of detail, with base models producing short, generic captions. This limitation persists even though VLMs are equipped with strong vision and language backbones. While supervised data and complex reward functions have been proposed to improve detailed image captioning, we identify a simpler underlying issue: a bias towards the end-of-sequence (EOS) token, which is introduced during cross-entropy training. We propose an unsupervised method to debias the model’s tendency to predict the EOS token prematurely. By reducing this bias, we encourage the generation of longer, more detailed captions without the need for intricate reward functions or supervision. Our approach is straightforward, effective, and easily applicable to any pretrained model. We demonstrate its effectiveness through experiments with three VLMs and on three detailed captioning benchmarks. Our results show a substantial increase in caption length and relevant details, albeit with an expected increase in the rate of hallucinations. 1 Introduction As vision-language models (VLMs) advance, new frontiers open up in multimodal research, some examples being the novel tasks of interactive visual dialogue ( Liu et al. 2023 ) and culturally-aware visual question answering ( Romero et al. 2024 ) . The fundamental task of image captioning ( Vinyals et al. 2015 ) has also seen an upgrade: in place of the classic benchmark MS COCO ( Lin et al. 2014 ) , a number of new evaluation datasets have been proposed, consisting of images with detailed (a.k.a fine-grained or dense) captions ( Onoe et al. 2024 ; Cho et al. 2022 ; Urbanek et al. 2024 ) . Detailed image captioning has extrinsic applications, e.g., assistive applications, as well as a key intrinsic application, as a form of image textualization for solving knowledge- and reasoning-intensive vision-language tasks using large language models ( Pi et al. 2024 ) . At its core, detailed image captioning does not require any new capabilities compared to coarse image captioning, i.e. both forms of the task rely on the ability of the vision encoder in a VLM to extract useful features from the input image, and of the LM backbone to decode the semantics of these features and express them in coherent text. Typically, the vision encoder and the language decoder in a VLM are initialized from extensively pretrained models, capable of complex feature extraction and long-form text generation, respectively. As such, detailed image captioning should be well within the reach of these components. Yet, during joint vision-language training the language decoder is trained to match the distribution of short coarse captions, thus losing the ability to generate long-form text. Various methods have been considered for training VLMs to generate detailed captions. State-of-the-art large VLMs are being extensively trained on instruction-tuning data for a range of tasks, including detailed image captioning ( Liu et al. 2023 ) . Smaller and more practical VLMs have also been trained specifically for this task, often using synthetic data induced from a larger VLM ( OpenAI 2023 ) . Away from direct supervision, some have experimented with reinforcement learning using a teacher VLM as a reward model ( Cho et al. 2022 ) , or sourcing distant supervision from downstream tasks ( Fisch et al. 2020 ) . In this work, we introduce a method for adapting pretrained VLMs to perform detailed image captioning without any form of supervision . We view the generation of detailed captions as a form of recovering existing model capabilities (rather than instilling new ones) and propose to achieve that through end-of-sequence (EOS) token debiasing. The EOS token exists in every single caption in the training data; combined with teacher forcing, this causes the model to have a bias towards predicting it. Elevating this bias should lead the model to generate longer text ( Newman et al. 2020 ; Kulikov et al. 2021 ) , and the conditioning on the image context during vision-language pretraining should ensure that this longer text remains grounded in the input image. Through our experiments, we show that simply pushing down the probability of the EOS token through sequence training of models trained on generic captions leads to longer, more detailed and largely accurate outputs. We conduct experiments on three popular VLMs: two variants of BLIP-2 ( Li et al. 2023 ) and PaliGemma ( Beyer et al. 2024 ) , chosen as they have not been explicitly trained for the task of detailed image captioning. Our finetuned models are evaluated against the base models on three dense captioning evaluation datasets. We find that across different models and metrics, our approach results in a substantial increase in the length of generated captions. Through a trade-off analysis of recall and hallucination metrics, we determine that the additional text in the caption provides relevant details, proving the effectiveness of sequence training with EOS debiasing as a way to induce detailed captions from pretrained VLMs. 2 Related work 2.1 Vision Language Modeling Here we introduce the common vision-language modeling framework as exemplified in the two VLM families included in our study: BLIP-2 ( Li et al. 2023 ) and PaliGemma ( Beyer et al. 2024 ) . Current approaches to vision-language modeling adopt an encoder-decoder approach, wherein input images are encoded into visual tokens, which are subsequently decoded into text. Both components are typically initialized with pretrained weights and exhibit powerful generalization capabilities. In order to learn a mapping of visual features into language semantics, the VLM is trained on image-text pairs. BLIP-2 is pretrained on image-caption pairs, and PaliGemma on a mixture of tasks: captioning, question-answering, OCR, object detection, and segmentation. 1 1 1 Note that this mixture of tasks does not include detailed image captioning: the data used for the captioning task consists of alt-text scraped from the web. While these multimodal pretraining datasets are very large indeed, they amount to a small fraction of the data used to train the language backbone of VLMs, and represent a considerably narrower distribution of mostly short texts. As such, the multimodal pretraining phase helps the language decoder to learn how to interpret visual features and map them into language, but it also shifts the distribution of the underlying language model in line with the text found in the multimodal training data. 2.2 Detailed Image Captioning Several works have attempted to improve the ability of VLMs to generate detailed image captions with fully supervised and weakly supervised approaches. We discuss these methods in terms of the type of supervision and training objectives they employ. Fully Supervised Methods rely on training datasets of images paired with detailed captions. The captions in these datasets are either human annotated ( Urbanek et al. 2024 ; Onoe et al. 2024 ) , with high annotation costs limiting the amount of data produced, or synthetically generated, using large closed-source VLMs ( Chen et al. 2023 ; Singla et al. 2024 ; Gaur et al. 2024 ) . Recent work shows increased hallucination and bias in models trained on synthetic captions ( Hirota et al. 2024 ) . Weakly Supervised Methods use no explicit training data for detailed image captioning, and instead