Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data

Haoxin Li, Boyang Li · 2025

Read Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data by Haoxin Li, Boyang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01167.txt.

Description

Li and Li make synthetic training pairs that teach vision-language models to notice small wording swaps. A model that truly grasps composition should tell apart people holding surfboards from people holding shovels. Their 2025 method, SPARCL, forges such near-twin image and caption pairs with a fast image generator steered by features from a real photo.

A style transfer pass then keeps the forged image close to the real one outside the changed spot. A self-adjusting loss drops suspect pairs and leans on hard, informative ones. Across four composition tests, training with the forged data lifted a standard model's average score by more than eight points. It also beat prior methods on three of the four tests. The authors frame forgery as cheap teaching data.

Catalog record

CREATORHaoxin Li, Boyang Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01167

Browse the full library · About the Tower of Babel