Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions

Jiarui Zhang ♢ , Ollie Liu ♢ , Tianyu Yu ♠ , Jinyi Hu ♠ , Willie Neiswanger ♢ ♢ University of Southern California, ♠ Tsinghua University · 2024

Read Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions by Jiarui Zhang ♢ , Ollie Liu ♢ , Tianyu Yu ♠ , Jinyi Hu ♠ , Willie Neiswanger ♢ ♢ University of Southern California, ♠ Tsinghua University (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08737.txt.

Introduction

Jiarui Zhang and co-authors train sight-tuned chat models on made-up shape worlds so they can copy what they see true. Chat models with eyes still fumble low sight facts: the lines, gaps, and tilts in a plain shape shot. Their 2024 study first builds a test set for that skill, then sweeps what helps. Top chat eyes fail the test by wide marks.

Help comes from rich fake shots and staged learn, where easy shapes lead hard ones. Staged food lets the eyes learn tasks they fail cold. Their Euclid kin, raised on fake shots alone, top the shape test and hold up on real shots too. Shape-true eyes could serve bots, scan reads, and make lines where a line off by a hair is a flaw.

Catalog record

CREATORJiarui Zhang ♢ , Ollie Liu ♢ , Tianyu Yu ♠ , Jinyi Hu ♠ , Willie Neiswanger ♢ ♢ University of Southern California, ♠ Tsinghua University
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08737

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions Jiarui Zhang ♢ , Ollie Liu ♢ , Tianyu Yu ♠ , Jinyi Hu ♠ , Willie Neiswanger ♢ ♢ University of Southern California, ♠ Tsinghua University Abstract Multimodal large language models (MLLMs) have made rapid progress in recent years, yet continue to struggle with low-level visual perception (LLVP)—particularly the ability to accurately describe the geometric details of an image. This capability is crucial for applications in areas such as robotics, medical image analysis, and manufacturing. In this paper, we first introduce Geoperception , a benchmark designed to evaluate an MLLM’s ability to accurately transcribe 2D geometric information from an image. Using this benchmark, we demonstrate the limitations of leading MLLMs, and then conduct a comprehensive empirical study to explore strategies for improving their performance on geometric tasks. Our findings highlight the benefits of certain model architectures, training techniques, and data strategies, including the use of high-fidelity synthetic data and multi-stage training with a data curriculum. Notably, we find that a data curriculum enables models to learn challenging geometry understanding tasks which they fail to learn from scratch. Leveraging these insights, we develop Euclid , a family of models specifically optimized for strong low-level geometric perception. Although purely trained on synthetic multimodal data, Euclid shows strong generalization ability to novel geometry shapes. For instance, Euclid outperforms the best closed-source model, Gemini-1.5-Pro, by up to 58.56% on certain Geoperception benchmark tasks and 10.65% on average across all tasks. = Website : euclid-multimodal.github.io = Model Weights & Datasets : huggingface.co/euclid-multimodal Code Repository : github.com/euclid-multimodal/Euclid 1 Introduction Multimodal large language models (MLLMs) have rapidly progressed in recent years, demonstrating remarkable potential in understanding and reasoning about the visual world through the powerful capabilities of large language models (LLMs) ( Liu et al. 2024c , Liu et al. 2024a , Achiam et al. 2023 , Team et al. 2023 , Hu et al. 2023 , Tong et al. 2024a , Wang et al. 2024a ) . These models have showcased strong performance in tasks such as visual question answering (VQA) ( Goyal et al. 2017 ) , image captioning ( Lin et al. 2014 ) , and multimodal reasoning ( Liu et al. 2023 ) . As one recent example, LLaVA-NeXT-34B ( Liu et al. 2024b ) achieves an impressive 83.7% accuracy on the VQAv2 benchmark ( Goyal et al. 2017 ) , a comprehensive benchmark on natural image question answering. While MLLMs achieve impressive results on tasks like VQA, their performance relies on high-level semantic extraction ( Tong et al. 2024b ) ; in contrast, they often fall short on low-level visual perception (LLVP)—i.e., the ability to accurately describe the geometric details of an image, such as the points, lines, angles, shapes, and spatial relationships among its constituent objects. This limitation becomes especially apparent in tasks requiring precise descriptions, such as mathematical visual problem solving ( Zhang et al. 2024a , Lu et al. 2023 ) , scientific visual understanding ( Yue et al. 2024 , Fu et al. 2024a ) , abstract visual reasoning ( Jiang et al. 2024 , Ahrabian et al. 2024 ) , and even simple visual comprehension ( Rahmanzadehgervi et al. 2024 , Wang et al. 2024b ) . For example, when interpreting a graph diagram, precise recognition of edges is essential for extracting reliable information, and in geometry problem-solving, accurate identification of relationships between line segments and points is fundamental ( Fu et al. 2024a ) . Beyond abstract tasks, LLVP is also vital in real-world applications, including spatial understanding for robotics, medical image analysis for accurate diagnosis, quality control in manufacturing to detect subtle defects, autonomous driving systems that rely on exact object localization or distance estimation, and augmented reality applications that demand precise overlay of virtual objects onto the real world. In this paper, we aim to study the challenges of LLVP in MLLMs, take steps to understand the root cause of their performance, and improve the models’ capabilities in this area. We begin by developing a benchmark dataset specifically designed to evaluate precise geometric perception, which we call Geoperception . As a focused test bed, this benchmark targets 2D geometry tasks. Using this benchmark, we demonstrate the limitations of leading closed and open MLLMs, followed by a comprehensive empirical study to explore strategies for significantly improving MLLM’s performance on geometric perception tasks. Our findings show the benefits of key factors such as model architecture, training techniques, and data strategies, including the use of synthetic data and multi-stage training with a data curriculum. Notably, we find that a data curriculum enables models to learn challenging geometry LLVP tasks, which they fail to learn from scratch, even when trained on a very large dataset. Using these lessons learned, we then train a family of models—using a carefully designed curriculum of synthetic data—that are specifically optimized for strong LLVP, which we call Euclid. We evaluate this family of models, and show that it excels on a variety of low-level geometric perception tasks. Our main technical contributions are as follows: • Geoperception Benchmark: We introduce a new benchmark dataset, Geoperception , derived from the Geometry-3K corpus ( Lu et al. 2021 ) , specifically designed to evaluate MLLMs’ ability to accurately perceive surface-level geometric information without requiring complex inference or reasoning. Our benchmark reveals shortcomings in precise geometric perception across all leading vision-language MLLMs, both closed and open-source. • Empirical Study and Synthetic Data Engine: To investigate the root cause of this performance, we conduct a detailed empirical exploration of MLLM architecture and training strategies. To aid in our investigation, we develop a synthetic data engine capable of generating high-fidelity visual descriptions of geometric elements. This study leads to key insights, such as the importance of certain architectural choices and the use of curriculum-based, multi-stage training with progressively more complex visual descriptions for improving low-level visual perception. • Euclid Model Family: Leveraging the insights from our exploration and our synthetic data engine, we train Euclid, a series of MLLMs tailored for high-quality geometric LLVP. Although purely trained on synthetic multimodal data with simple geometry shapes, Euclid generalizes strongly to the real-world geometry images from Geoperception benchmark, for instance, outperforming the best closed-source model, Gemini-1.5-Pro, by up to 58.56% on certain benchmark tasks and 10.65% across the tasks. 2 Background and Related Work We provide an overview of prior efforts that assess and improve low-level perception and geometric reasoning in MLLMs, and highlight our contributions in data synthesis, evaluation, and training. Vision-Language MLLMs. While recent iterations of LLMs feature a standardized model architecture and pretraining recipe, MLLMs still often differ in design choices for infusing visual inputs. One popular design is to align continuous visual features with the embedding space of a backbone LLM ( Liu et al. 2024a , Liu et al. 2024b , Dubey et al. 2024 , McKinzie et al. 2024 , Tong et al. 2024a , Beyer et al. 2024 , AI 2023 , Wang et al. 2024a ) ; another approach involves tokenizing visual inputs to be trained jointly with language tokens ( Team et al. 2023 , Team 2024a ) . These modules are often infused with a decoder-only LLM, but others have explored

Read the complete text