Visual Lexicon: Rich Image Features in Language Space

XuDong Wang, Xingyi Zhou, Alireza Fathi, Trevor Darrell, Cordelia Schmid · 2024

Read Visual Lexicon: Rich Image Features in Language Space by XuDong Wang, Xingyi Zhou, Alireza Fathi, Trevor Darrell, Cordelia Schmid (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06774.txt.

Introduction

Turning a picture into words usually loses either the broad meaning or the fine detail. Wang and co-authors present Visual Lexicon, a set of learned tokens that live in the same space as text yet carry rich image content. The tokens are trained so that a frozen text-to-image model can rebuild the original picture from them, which forces them to keep both the scene's sense and its small features.

A single token already rebuilds images more faithfully than text embeddings do. The tokens also personalize image generation in a zero-shot way and, used as a vision encoder, lift a strong baseline across fifteen benchmarks. They can mix freely with ordinary words in prompts.

Catalog record

CREATORXuDong Wang, Xingyi Zhou, Alireza Fathi, Trevor Darrell, Cordelia Schmid
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06774

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visual Lexicon: Rich Image Features in Language Space XuDong Wang, Xingyi Zhou, Alireza Fathi, Trevor Darrell, Cordelia Schmid Abstract We present Visual Lexicon, a novel visual language that encodes rich image information into the text space of vocabulary tokens while retaining intricate visual details that are often challenging to convey in natural language. Unlike traditional methods that prioritize either high-level semantics (e.g., CLIP) or pixel-level reconstruction (e.g., VAE), ViLex simultaneously captures rich semantic content and fine visual details, enabling high-quality image generation and comprehensive visual scene understanding. Through a self-supervised learning pipeline, ViLex generates tokens optimized for reconstructing input images using a frozen text-to-image (T2I) diffusion model, preserving the detailed information necessary for high-fidelity semantic-level reconstruction. As an image embedding in the language space, ViLex tokens leverage the compositionality of natural languages, allowing them to be used independently as “text tokens” or combined with natural language tokens to prompt pretrained T2I models with both visual and textual inputs, mirroring how we interact with vision-language models (VLMs). Experiments demonstrate that ViLex achieves higher fidelity in image reconstruction compared to text embeddings—even with a single ViLex token. Moreover, ViLex successfully performs various DreamBooth tasks in a zero-shot, unsupervised manner without fine-tuning T2I models. Additionally, ViLex serves as a powerful vision encoder, consistently improving vision-language model performance across 15 benchmarks relative to a strong SigLIP baseline. Figure 1 : Given the cute corgi painting in the top left corner, how can we extract a visual representation that captures semantic-level information – such as object categories and layouts – while preserving rich visual details like image styles, textures and colors? We introduce ViLex model that generates image representations in the text vocabulary space , acting as a new visual “language”, while retaining intricate visual details that are difficult, if not impossible, to convey in natural language. The set of images (generated under different diffusion noises) in the 2 \times 2 grid, which are highly semantically and visually similar to each other, is created by using ViLex as “text” prompts for text-to-image diffusion models. 0 0 footnotetext: ​​​​​​​​​​​​*Work done in a Google DeepMind internship. ✉{xudongwang, zhouxy}​@​google.com​​ 1 Introduction How should we represent an image? This is a fundamental question in computer vision. Over decades of progress, there have been two primary approaches: representations optimized for understanding high-level semantics [ 14 , 53 , 50 ] , or for high-fidelity image reconstruction [ 23 , 34 , 86 ] , often used in image generation [ 60 , 55 ] . Understanding-focused models like CLIP [ 53 ] and DINO [ 8 ] capture high-level semantics but lose pixel-level details. Conversely, reconstruction-focused models, such as VAEs [ 34 ] , retain fine visual details but lack semantic richness, making them less effective for tasks like vision-language modeling. In this paper, we aim to address the question: “can a single representation excel in both image reconstruction and semantic understanding?” Figure 2 : top) ViLex empowers linguistic space to capture visual richness . We propose ViLex , an image encoder that maps images into the vocabulary space, effectively preserving semantic information and intricate visual details. The embeddings from ViLex function as a Visual Lexicon that preserve semantic and intricate visual details of the image. ViLex is trained with a frozen text-to-image diffusion model and can be utilized independently as “text” tokens for image generation. bottom) Linguistic space empowers ViLex to enjoy compositionality . ViLex can be combined with natural language tokens for prompting a pretrained T2I diffusion models with both visual and textual cues. To bridge this gap, we introduce ViLex that encodes images into a Vi sual Lex icon within the text space. ViLex model is designed to capture both high-level semantics – such as object categories and spatial layouts – while preserving rich visual details like styles, and textures that are difficult or even impossible to articulate in natural language. We achieve this, as illustrated in Fig. 2 , by leveraging a self-supervised learning strategy based on a frozen, pretrained text-to-image (T2I) diffusion model, which acts as the source of supervisory signals. Although initially developed for generative purposes, many recent works [ 3 , 35 , 19 , 79 ] have discovered that diffusion models [ 60 , 56 , 27 ] inherently capture both semantic and detailed visual information through their denoising process. To incorporate rich visual information into our ViLex model, we repurpose diffusion models as decoders within an autoencoder [ 25 , 34 , 4 , 73 ] framework. ViLex embeddings are mapped into the latent space of the T2I diffusion model’s vocabulary tokens—specifically, the index-to-embedding lookup matrix of the text encoder [ 53 , 54 ] , which converts text token IDs into embeddings. Using diffusion models as decoders for semantic-level image reconstruction , rather than traditional VAE decoders [ 34 ] or MAE [ 23 ] designed for pixel-level reconstruction , encourages the model to learn meaningful semantic representations that are highly transferable to diverse visual scene understanding tasks. This design enables ViLex to harness the rich visual knowledge embedded in diffusion models while maintaining a lightweight structure, making it well-suited for a broad range of understanding tasks beyond diffusion models’ original generative applications. The ViLex model consists of a vision encoder that extracts visual representations from the input image and an attention pooling layer that transforms the visual representation into visual lexicon tokens. During training, ViLex model is optimized with an image reconstruction loss, receiving gradients from the frozen diffusion model and its text encoder to fine-tune the visual lexicon tokens for accurately reconstructing images with similar appearance. Additionally, we propose the TailDrop strategy during training, where the last k visual lexicon tokens are randomly dropped to encourage the earlier tokens to encapsulate richer semantic information. During inference, the number of tokens can be dynamically adjusted to meet user requirements. Our ViLex model is designed to support both image generation and understanding tasks. For image generation: ViLex tokens can be directly used as “text-prompts”, enabling the re-creation of semantically and visually similar images. Experiments on COCO image reconstruction demonstrate that our ViLex significantly outperforms its counterparts image-guided DALL \cdot E 3 [ 5 ] and DeDiffusion [ 76 ] in terms of the layout, semantic, and style consistency with the input image, based on human studies. Notably, even with just a single ViLex token, the FID score of ViLex remains lower than that of DeDiffusion [ 76 ] , showcasing the representational power of Visual Lexicon. Additionally, ViLex embeddings can seamlessly integrate with text prompts, for example, “an image similar to [ViLex tokens], in Van Gogh style” , enabling multimodal image generation and DreamBooth [ 58 ] tasks in a zero-shot fashion by prompting a frozen T2I model with both visual and textual inputs. For image understanding: replacing the strong semantic-pretrained backbone SigLIP [ 88 ] with ViLex’s vision encoder in vision-language models [ 6 ] leads to improvements across various vision-language tasks, including image and video captioning [ 10 , 78 ] , visual question answering [ 21 ] , and referring segmentation [ 83 ] . The main contributions of our work are: • We propose ViLex, an image encoder that maps images into the text space of

Read the complete text