DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding

Zhihang Zhong, Xiao Sun · 2024

Read DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding by Zhihang Zhong, Xiao Sun (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01115.txt.

Description

Hao Wu, Zhihang Zhong, and Xiao Sun proposed in 2024 a retrieval-augmented method for image captioning called DIR. Caption models often fail on new datasets unlike their training data. Retrieval can help, but image features tuned to one caption per image carry annotator bias, and retrieved text is often used too shallowly.

Their first fix guides image feature learning with a pretrained diffusion model that rebuilds noisy images, so features hold richer visual detail. The second fix is a high-quality retrieval database that gives fuller semantic context for writing the caption. Across tests, DIR kept strong in-domain scores and clearly improved out-of-domain captioning, with no added inference cost.

Catalog record

CREATORZhihang Zhong, Xiao Sun
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01115

Browse the full library · About the Tower of Babel