SLIP: Structural-aware Language-Image Pretraining for Vision-Language Alignment
Wenbo Lu · 2025
Read SLIP: Structural-aware Language-Image Pretraining for Vision-Language Alignment by Wenbo Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03019.txt.
Introduction
Wenbo Lu argues that image-text training wastes the web of links between products, posts, and people. Standard vision-language pretraining treats each pair as an island. Yet domains like online shopping carry rich relational structure, such as graphs of items bought together. In a 2025 paper, Lu introduces SLIP, a pretraining method that learns from those links as well as the pairs.
SLIP adds a structural contrastive loss that aligns images and text while modeling ties between neighboring entities in a graph. To train it, Lu built a large Amazon co-purchase data set with product images and text. SLIP beat the well-known CLIP model on retrieval and classification, in both zero-shot and few-shot tests. The result credits relational supervision, inspired by how human memory maps knowledge as relations.
Catalog record
| CREATOR | Wenbo Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-03019
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SLIP: Structural-aware Language-Image Pretraining for Vision-Language Alignment Wenbo Lu Abstract Vision–Language Pretraining (VLP) has achieved remarkable success across various downstream tasks, but such gains are largely driven by scaling up on training data. Yet, literature methods treat image-text pairs as isolated training examples; this neglects the rich relational structure naturally present in many domains, such as e-commerce product co-purchase graphs and social recommendation networks. Inspired by neuroscientific evidence that human encodes knowledge as relationship cognitive maps, we introduce S tructure-aware L anguage- I mage P retraining (SLIP) . SLIP integrates a structural contrastive loss to align modalities while also modeling relationships between neighboring entities in a structured graph. To support this paradigm, we construct a large-scale Amazon Product Co-purchase Multimodal Graph Dataset , enabling structured cross-modality supervision at scale. Experiment results show that SLIP consistently outperforms CLIP on cross-modal retrieval and classification tasks in both zero-shot and few-shot settings, showing the value of relational supervision for cross-modal alignment. 1 Introduction Vision-language alignment has emerged as a key challenge in multimodal representation learning, with recent pretraining approaches achieving remarkable success by learning from web-scale data, driving progress in multimodal tasks such as image-text retrieval, visual question answering (VQA), and image captioning 6 . Ground-breaking work CLIP ( 15 ) has shown that a simple contrastive objective can yield state-of-the-art representations when scaled to millions of noisy image-text pairs, and such large-scale training has thus become the paradigm for vision-language foundation models. However, these web-scale corpora are notoriously noisy : captions can be generic, off-topic, or mismatched to the image. As a result, the performance is capped by the suboptimal source of supervision. Prior works addressing this challenge generally fall into two main categories: (1) exploring alternative forms of supervision within the data—for example, 24 introduces token-level alignment—and (2) refining existing supervising labels through techniques like knowledge distillation or bootstrapped relabeling, as in BLIP ( 10 ) . While these approaches enhance representation quality, they share a common limitation: they treat each image–text pair in isolation, overlooking the rich structural relationships among entities that are inherent in many real-world datasets. Many domains exhibit strong underlying structure. In e-commerce, products form co-purchase graphs-“customers who bought X also bought Y”-while in vision datasets, images are often connected via shared attributes or hierarchical semantics (e.g., “lion” and “tiger” as big cats). Similarly, in knowledge bases, entities are arranged into richly annotated relational graphs. Yet, existing VLP models ignore these structures and instead learn from contrasting loosely related individual pairs, losing consistency and semantic grounding. Leveraging structured relations could help capture latent connections and contextualize visual and textual concepts beyond what any single caption can provide. This idea finds support in cognitive neuroscience. Humans do not store knowledge in isolation; rather, they organize it into cognitive maps -internal graph-like representations that encode relations among entities and facilitate flexible reasoning ( 1 ) . These maps enable us to generalize across experiences by exploiting structured relationships: when shown a laptop, for example, a shopper may automatically retrieve related items like a charger or a mouse. Representational distances in the brain are shaped not just by feature similarity but by relational proximity in learned conceptual spaces. Inspired by this, we posit that relational structure should likewise inform how machines align visual and textual modalities. In this paper, we introduce S tructure-aware L anguage- I mage P retraining (SLIP) , a framework that augments vision-language contrastive learning with structural supervision. SLIP assumes that image-text pairs are nodes in a relational graph, where edges reflect semantic or contextual proximity-such as co-purchase, co-view, or hierarchical links. Specifically, Contextual signals are first aggregated using modality-specific Graph Attention Network (GAT) layers and then fused to form a unified node representation. This representation is used in a structural contrastive loss , which encourages embeddings of graph-neighboring nodes to be close in the space. This encourages cross-modal alignment for each pair, and coherent placement of related items, so that learned representations can reflect alignment and relational proximity. To support research in structure-aware multimodal learning, we curate and release the Multimodal Amazon Product Co-purchase Graph Dataset , a large-scale dataset of image-text pairs connected via real-world co-purchase edges. The dataset spans a wide range of product categories, with each product node annotated with both an image and a textual title or description. Edges represent purchase patterns mined from user behavior, offering weak but meaningful supervision. This dataset serves as a realistic and scalable benchmark for structured vision-language representation learning, where both multimodal data and relational graphs are available. Contributions. Our work makes the following key contributions: • We propose SLIP, a simple and scalable framework for incorporating structural supervision into contrastive vision-language pretraining. SLIP introduces a structural contrastive objective that regularizes image-text embeddings based on instance-level graph connectivity. • We introduce the Amazon Product Co-purchase Multimodal Graph, a new large-scale dataset of image-text pairs linked by relational structure. It offers a challenging and realistic test bed for evaluation. • We demonstrate through extensive experiments that SLIP achieves state-of-the-art results on retrieval and classification tasks in few-shot settings, outperforming standard CLIP baselines and providing empirical evidence for the value of structural context. 2 Related Work Vision-Language Pre-training. Vision-language pre-training learns strong generalization over various downstream tasks. Over the years, several model architectures have been proposed to support different types of applications. Dual-encoder architectures, such as CLIP ( 15 ) and ALIGN ( 9 ) , encode images and texts separately and align them in a shared embedding space using contrastive learning. Fusion-encoder models like ViLBERT ( 17 ) and UNITER ( 3 ) instead perform deep cross-modal interactions by jointly encoding vision and language inputs. Encoder-decoder architectures, including VL-T5 ( 4 ) , SimVLM ( 21 ) , and BLIP ( 10 ) , follow a generative modeling approach that decodes text conditioned on visual features. More recently, unified transformer architectures such as PaLI ( 2 ) and Uni-Perceiver v2 ( 20 ) aim to support a wide variety of vision and vision-language tasks using a single, flexible framework. Alongside these architectural advances, pre-training objectives have also converged on a few key formulations: image-text contrastive learning ( 15 ; 26 ; 10 ) , image-text matching ( 11 ; 19 ) , and masked modeling objectives applied to language or vision tokens ( 27 ; 20 ) . However, these methods overlook the relational structure that exists among entities in many real-world domains, which SLIP explores. Scene-graph for object-level alignment. Early work injected intra‑image structure by coupling object graphs with captioners or retrieval models. GCN-LSTM ( 25 ) and VSRN ( 12 ) propagated object–object interactions to improve description and matching accuracy. Follow‑ups for grounding and VQA ( 23 ; 12 ) showed analogous gains for referring-expression grounding by jointly scene-graphing image regions and linguistic phrases. More recently, ERNIE‑ViL ( 28 ) and Structure-CLIP ( 8 ) pushed this idea to the pre-training stage, either by multi‑task scene‑graph prediction or by scene‑graph‑driven hard negatives on top of CLIP. These models innovate by exploiting object interactions, yet they remain constrained to individual image-caption pairs and require costly scene-graph annotations. Instead, SLIP operates on the instance-level graphs defined over the dataset (e.g., co-purchase graphs), requiring no object detection or