ReflectCAP: Detailed Image Captioning with Reflective Memory

Kyungmin Min, Minbeom Kim, Kang-il Lee, Seunghyun Yoon, Kyomin Jung · 2026

Read ReflectCAP: Detailed Image Captioning with Reflective Memory by Kyungmin Min, Minbeom Kim, Kang-il Lee, Seunghyun Yoon, Kyomin Jung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12357.txt.

Introduction

Min and four colleagues present ReflectCAP, reflective note-guided captioning that tackles the tension between factual grounding and fine-grained coverage in detailed image captions. A multi-agent pipeline distills what the target vision-language model hallucinates and overlooks into reusable structured reflection notes. At inference the notes steer the model on both axes, away from known errors and toward overlooked details, jointly improving factuality and coverage. Across eight models it reaches the Pareto frontier on the trade-off and beats model scaling on the quality-cost balance.

Catalog record

CREATORKyungmin Min, Minbeom Kim, Kang-il Lee, Seunghyun Yoon, Kyomin Jung
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58328
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12357

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ReflectCAP: Detailed Image Captioning with Reflective Memory Kyungmin Min, Minbeom Kim, Kang-il Lee, Seunghyun Yoon, Kyomin Jung Abstract Detailed image captioning demands both factual grounding and fine-grained coverage, yet existing methods have struggled to achieve them simultaneously. We address this tension with Reflective Note-Guided Captioning (ReflectCAP), where a multi-agent pipeline analyzes what the target large vision-language model (LVLM) consistently hallucinates and what it systematically overlooks, distilling these patterns into reusable guidelines called Structured Reflection Notes. At inference time, these notes steer the captioning model along both axes—what to avoid and what to attend to—yielding detailed captions that jointly improve factuality and coverage. Applying this method to 8 LVLMs spanning the GPT-4.1 family, Qwen series, and InternVL variants, ReflectCAP reaches the Pareto frontier of the trade-off between factuality and coverage, and delivers substantial gains on CapArena-Auto, where generated captions are judged head-to-head against strong reference models. Moreover, ReflectCAP offers a more favorable trade-off between caption quality and compute cost than model scaling or existing multi-agent pipelines, which incur 21–36% greater overhead. This makes high-quality detailed captioning viable under real-world cost and latency constraints. 1 Introduction Hyper-detailed captions capture not only salient objects but also their attributes, orientations, spatial relations, background context, and subtle visual states, forming a comprehensive textual representation of an image. Such captions have become a key ingredient in downstream multimodal systems—improving prompt fidelity for text-to-image and text-to-video generation, and serving as a reasoning aid for compositional and grounded decision-making [ 1 , 9 , 24 , 2 , 12 , 8 ] . Large vision-language models (LVLMs) can produce these descriptions fluently [ 21 , 41 , 5 , 20 ] , yet they frequently hallucinate—generating text that is not grounded in the image. This limitation is widely attributed to the tendency of language priors to progressively dominate over visual evidence as generation length increases, leading the model to describe what is statistically probable rather than what is actually depicted [ 25 , 16 , 15 , 19 ] . Hyper-detailed captioning, which inherently requires such extended generation, thus remains a critical bottleneck for reliable deployment. The straightforward remedy is supervised fine-tuning on human-authored detailed captions [ 8 , 26 ] to improve LVLMs’ intrinsic performance; however, as we demonstrate in Section 5.1 , such captions often exceed the model’s perceptual capacity, even increasing hallucinations well beyond the base model. Moreover, this approach requires not only expensive human annotation but also additional training, further limiting its practicality. An alternative is inference-time correction, where the model iteratively revises its own output without additional training—an approach that has proven effective for LLMs [ 13 , 22 ] . However, recent studies show that LVLMs struggle to self-correct during inference without external feedback, as they tend to confirm rather than rectify their own errors [ 10 , 38 ] . Moreover, iterative revision lengthens the text context, further amplifying language-prior reliance over visual evidence [ 25 , 16 ] . Taken together, these limitations suggest that a single LVLM alone is unlikely to fully address the detail–faithfulness tension, and that external guidance from a multi-agent system is needed. Figure 1 : Overview of ReflectCAP. In the offline phase, a multi-agent reflective learning pipeline distills a target LVLM’s recurring captioning errors and omissions into Structured Reflection Notes. In the online phase, these notes guide caption generation for new images, producing captions that better balance factuality and coverage. To this end, we propose ReflectCAP (Reflective Note-Guided Captioning), a gradient-free framework that distills a target LVLM’s recurring errors into an agentic memory called Structured Reflection Notes and leverages them for inference-time steering (Figure 1 ). ReflectCAP operates in two distinct phases. In the offline phase, a multi-agent pipeline critiques the target model’s captions against a small set of human-annotated references to diagnose its systematic error patterns, separately encoding 1) hallucination patterns and 2) omission patterns into generalized guideline notes. In the online phase, each set of notes serves a distinct role in steering generation: one produces a grounded base caption that suppresses recurring hallucinations, and the other produces a detail-focused caption covering typically neglected visual elements. A final merge step combines both with the image as a reference, producing a comprehensive caption that improves factuality and coverage simultaneously. Empirically, ReflectCAP sets new pareto frontier under the factuality–coverage evaluation proposed by Lee et al. [ 16 ] , substantially expanding coverage while preserving precision and achieving the highest F1 across model families. This improvement also holds on CapArena-Auto [ 3 ] , a pairwise benchmark that evaluates captions holistically and is closely aligned with human preference. On 600 images evaluated with CapArena-Auto, ReflectCAP yields substantial win-rate improvements of +32.2 and +21.9 points over the corresponding baselines for the GPT and open-source model families, respectively. Furthermore, ReflectCAP can elevate smaller models beyond flagship baselines; for instance, GPT-4.1-mini with our method surpasses GPT-5.2 in CapArena-Auto win rate. Beyond performance comparisons, ReflectCAP consistently offers a more compute-efficient path to improving caption quality than either scaling model size or scaling inference-time computation; for example, InternVL3.5-4B with ReflectCAP achieves a comparable factuality–coverage F1 to InternVL3.5-38B while requiring approximately 8 times lower inference TFLOPs, and outperforms existing multi-agent pipelines while incurring 21–36% lower compute overhead, suggesting that ReflectCAP can serve as a practical, cost-effective alternative to both model scaling and inference-time computation scaling for detailed captioning. 2 Related Work 2.1 Detailed Image Captioning Obtaining large-scale detailed image captions is increasingly important, as they serve as training signals for text-to-image and text-to-video generation and as reasoning aids for compositional vision–language tasks [ 1 , 9 , 2 , 8 ] . Approaches to scaling detailed captions broadly follow two directions. The first relies on human-authored dense captions (e.g., DCI [ 33 ] , DOCCI [ 26 ] , IIW [ 8 ] ), which offer high fidelity and coverage but are prohibitively expensive to scale beyond limited datasets. The second approach employs LVLMs as automated captioners. However, in long-form generation, these models often over-rely on language priors, which results in hallucinated but unsupported details [ 25 , 15 ] and the omission of subtle visual attributes [ 7 , 28 , 23 ] . Many existing mitigation methods primarily improve factuality (precision) without comparably improving descriptive coverage (recall), leaving the two principal quality axes of detailed captioning in tension [ 40 , 17 , 11 , 6 , 35 , 42 ] . Our work addresses this trade-off by distilling reflection notes that explicitly capture recurring hallucination patterns and missing-detail patterns, enabling separate control of hallucination suppression and detail recovery at inference time. 2.2 Reflective Memory in Agentic Frameworks Recent advancements in LLM-based agents have successfully leveraged reflective memory to refine behavior in long-horizon tasks. By analyzing historical trajectories and inference-time feedback, these text-based models synthesize reusable reasoning strategies and deploy them at appropriate moments to guide subsequent multi-step decision-making [ 32 , 43 , 30 , 34 , 27 ] . However, extending this long-horizon reflection paradigm to LVLMs is fundamentally problematic. Unlike pure text generation, LVLMs struggle to reliably extract error patterns over extended reasoning chains [ 10 , 38 ] . As the number of inference steps increases, the visual evidence becomes increasingly diluted, and the models fall prey

Read the complete text