Learning Visual Composition through Improved Semantic Guidance

Austin Stone, Hagen Soltau, Robert Geirhos, Xi Yi, Bingyi Cao, Kaifeng Chen, Abhijit Ogale, Jonathon Shlens · 2024

Read Learning Visual Composition through Improved Semantic Guidance by Austin Stone, Hagen Soltau, Robert Geirhos, Xi Yi, Bingyi Cao, Kaifeng Chen, Abhijit Ogale, Jonathon Shlens (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15396.txt.

Introduction

Austin Stone and co-authors wrote a paper from 2024 on teaching computers that a scene is made of parts in relation. Most image tools learn a photo as a loose bag of things, not as things acting on each other. That gap shows when a tool must tell who chases whom, or what sits on what. Past fixes built special model shapes for the task. This team took a plainer path.

They enriched the text lines that come with training photos. A standard image-text model was then trained on the richer data with no shape change. On hard tests of scene make-up, past models scored near chance. The enriched model scored far higher and beat all the special-shape rivals. Tests that removed parts of the fix showed the richer lines did the work. The model also did well at finding the right photo for a text query.

Catalog record

CREATORAustin Stone, Hagen Soltau, Robert Geirhos, Xi Yi, Bingyi Cao, Kaifeng Chen, Abhijit Ogale, Jonathon Shlens
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15396

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Visual Composition through Improved Semantic Guidance Austin Stone, Hagen Soltau, Robert Geirhos, Xi Yi, Bingyi Cao, Kaifeng Chen, Abhijit Ogale, Jonathon Shlens Abstract Visual imagery does not consist of solitary objects, but instead reflects the composition of a multitude of fluid concepts. While there have been great advances in visual representation learning, such advances have focused on building better representations for a small number of discrete objects bereft of an understanding of how these objects are interacting. One can observe this limitation in representations learned through captions or contrastive learning – where the learned model treats an image essentially as a bag of words. Several works have attempted to address this limitation through the development of bespoke architectures. In this work, we focus on simple and scalable approaches. In particular, we demonstrate that by improving weakly labeled data, i.e. captions, we can vastly improve the performance of standard contrastive learning approaches. Previous CLIP models achieved near chance rate on challenging tasks probing compositional learning. However, our simple approach boosts performance of CLIP substantially and achieves state of the art results on compositional benchmarks such as ARO and SugarCrepe. Furthermore, we showcase our results on a relatively new captioning benchmark derived from DOCCI. We demonstrate through a series of ablations that a standard CLIP model trained with enhanced data may demonstrate impressive performance on image retrieval tasks. 1 Introduction Figure 1 : Summary of results. Left: Previous state-of-the-art results in multimodal embeddings have limited understanding of the composition of images [ 26 , 14 ] . Right: The goal of this work is to learn multimodal embeddings which reflect a strong understanding of the composition of visual and semantic information. Images and captions are red and blue, respectively. Visual composition is a critical problem. Learning representations that capture how attributes and relationships of objects are represented is a critical task for any system that attempts to summarize the multitudes of concepts imbued within pixel space. Many SOTA multimodal models do not learn embeddings that have an understanding of the relationship of objects and descriptions [ 22 , 36 ] . One reason for this failure is that the visual representations are not rich enough to capture the interactions and relationships between various parts of an image or video. Current approaches have focused on building new bespoke architectures that leverage multi-task learning through multiple losses or side data [ 17 , 37 ] . Although these models achieve SOTA results on visual composition tasks [ 36 ] , it is unclear how scalable these approaches are given limitations on gathering high quality, specialized side information or the complexity of the architectures. Instead, we search for scalable solutions that focus on simplicity. The goal of this work is to employ a basic multimodal model solely employing a single contrastive learning objective [ 26 , 14 ] , and attempt to improve the learned representations with minimal changes. We posit that underlying visual architecture (e.g. ViT [ 8 ] ) contains sufficient parameters and scale to capture visual composition. Furthermore, we posit that contrastive learning is a sufficient training signal for multimodal learning so long as the target semantic embedding is rich enough. In particular, we ask if enriching the target semantic embedding is entirely sufficient to make a contrastive learning model capture visual composition. We answer this question by identifying a small set of minimal changes to improve the target semantic embedding of a model. First, we recaption our training data using a strong multimodal foundation model (Figure 2 . Second, we replace the text tower for CLIP [ 26 , 14 ] with a powerful text-based foundation model. We find that these two changes are entirely sufficient to greatly improve the visual embedding representation. We measure this effect using standard benchmarks [ 36 ] and discuss more newer benchmarks [ 22 , 28 ] to showcase these performance gains. Notably, we find that for detailed image retrieval tasks, our approach improves the open-source CLIP model [ 26 ] recall @1 from 58.4% to 94.5%. In summary, we focus on building simple and scalable techniques for improving compositionality in learned multimodal models focused on the problem of retrieval. We summarize our main contributions as follows: • Rich visual captions may be automatically generated using foundational models with simple techniques to minimize hallucinations (Sec. 2 , 3.3 , and 3.5 ). • Rich visual captions are sufficient for correcting many of the failures in compositional representation (Sec. 3.1 ). • COCO captioning benchmark [ 19 ] for measuring visual representations is saturated and subject to overfitting (Sec 3.2 ). New benchmarks are necessary for measuring improvements [ 22 , 36 ] (Sec 3.3 ). 2 Methods Figure 2 : Summary of methodology. All images were recaptioned using a multimodal foundation model grounded on the image and the alt-text for the image on the web page. In this case, we show captions generated using Gemini 1.5 Flash [ 30 ] . We highlight aspects of the new caption in blue that leverage the alt-text or demonstrate a capability. Note how the generated captions leverage information provided by the alt-text or perform OCR on the original image to improve the caption. Captions are for images from CC-12M [ 3 ] . Figure 3 : Synthetic negative captions. We prompt a foundation model [ 30 ] to generate 64 million synthetic positive and negative annotations. To generate the negative prompts, we provide few shot examples matching the style of ARO relations and attributes evaluation. Captions are for images from CC-12M [ 3 ] . Our method builds on the CLIP architecture proposed in [ 26 , 13 ] . We introduce two key modifications: 2.1 Semantic guidance with grounded recaptioning We start with an English-only subset of the WebLI dataset [ 4 , 5 ] consisting of 1B high quality images paired with freeform text scraped from the corresponding alt-text within a web page. We posit that the original alt-text is noisy and a limiting factor in the performance of an multimodal model [ 16 ] . Thus, we focus on creating a new set of captions to improve the alt-text . To enhance caption quality, we leverage Gemini 1.5 Flash [ 27 ] to generate new captions. In early experiment, we provide the original image as well as the alt-text and web page title, and prompt the model to generate a new caption that describes the underlying image. We iterated on the prompt (see Appendix for details) to arrive at a method that minimizes hallucinations while providing rich descriptions. Figure 2 summarizes our simple procedure. These examples showcase features in the recaptioning procedure. First, the grounding text supplied by the web page title and alt-text provides rich grounding information which can be exploited by the model. Second, the multimodal model can employ OCR in the original image to improve the noisy alt-text and correct for errors. In early experiments, we additionally found significant gains by providing two forms of data augmentation. First, we performed sentence sampling, where by randomly selected subsets of caption sentences are used as targets for the model. Second we synthesized 2 million “hard negative” examples to our training mixture (Figure 3

Read the complete text