LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perception
Yuan-Hong Liao, Sven Elflein, Liu He, Laura Leal-Taixé, Yejin Choi, Sanja Fidler, David Acuna · 2025
Read LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perception by Yuan-Hong Liao, Sven Elflein, Liu He, Laura Leal-Taixé, Yejin Choi, Sanja Fidler, David Acuna (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15362.txt.
Introduction
Liao and a team including Acuna build a data set for long visual reasoning in twenty twenty-five. Recent models gain a lot from long chains of thought on math and code tasks. For tasks that need looking at images, such long thought is less explored. Quick, shallow answers often seem to be enough there.
Their data set holds thirty thousand long thought traces for perception tasks. They build it in three steps, starting from detailed image descriptions. First they make checkable multiple choice questions, then they get short chains from vision models. Frontier models then expand those chains into full long thoughts. A tuned seven billion parameter model trained on the data improved on five vision tests. It also gained on a text reasoning test.
Catalog record
| CREATOR | Yuan-Hong Liao, Sven Elflein, Liu He, Laura Leal-Taixé, Yejin Choi, Sanja Fidler, David Acuna |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15362
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perception Yuan-Hong Liao, Sven Elflein, Liu He, Laura Leal-Taixé, Yejin Choi, Sanja Fidler, David Acuna Abstract Recent reasoning models through test-time scaling have demonstrated that long chain-of-thoughts can unlock substantial performance boosts in hard reasoning tasks such as math and code. However, the benefit of such long thoughts for system-2 reasoning is relatively less explored in other domains such as perceptual tasks where shallower, system-1 reasoning seems sufficient. In this paper, we introduce LongPerceptualThoughts , a new synthetic dataset with 30K long-thought traces for perceptual tasks. The key challenges in synthesizing elaborate reasoning thoughts for perceptual tasks are that off-the-shelf models are not yet equipped with such thinking behavior and that it is not straightforward to build a reliable process verifier for perceptual tasks. Thus, we propose a novel three-stage data synthesis framework that first synthesizes verifiable multiple-choice questions from dense image descriptions, then extracts simple CoTs from VLMs for those verifiable problems, and finally expands those simple thoughts to elaborate long thoughts via frontier reasoning models. In controlled experiments with a strong instruction-tuned 7B model, we demonstrate notable improvements over existing visual reasoning data-generation methods. Our model, trained on the generated dataset, achieves an average +3.4 points improvement over 5 vision-centric benchmarks, including +11.8 points on V ∗ Bench. Notably, despite being tuned for vision tasks, it also improves performance on the text reasoning benchmark, MMLU-Pro, by +2 points. 1 1 1 Project website: https://andrewliao11.github.io/LongPerceptualThoughts 0 0 footnotetext: ♠ University of Toronto, Vector Institute, ♢ NVIDIA, ♣ Purdue University 1 Introduction Reasoning models, such as OpenAI’s o1 ( OpenAI et al., 2024 ) and Deepseek’s R1 ( DeepSeek-AI et al., 2025 ) , have demonstrated remarkable capabilities in solving complex reasoning problems by scaling test-time compute. Intuitively, they increase the number of tokens generated at inference-time, allowing the model to “think longer”—producing longer chain-of-thoughts (CoTs) that go beyond typical linear rationales that mimic textbook responses. However, despite numerous attempts to match the performance of models like o1 or R1 on challenging math benchmarks ( MAA, 2024 ; Lightman et al., 2023 ) , less effort has been directed toward tasks beyond mathematical reasoning. Most notably, how to generate synthetic, long-form CoT reasoning traces that solve vision-centric tasks —suitable for both supervised fine-tuning and preference-based RL—remains largely underexplored. Vision-centric tasks have proven challenging for vision-language models (VLMs), especially when the tasks require object counting and localization, scene understanding, and 2D/3D spatial reasoning ( Liao et al., 2024a ; Rahmanzadehgervi et al., 2024 ; Campbell et al., 2024 ) . Prior works have addressed these challenges by helping VLMs “see” better. Common approaches include modifying the input image ( e.g ., through cropping) or incorporating intermediate representations, into the CoT ( Wu & Xie, 2024 ; Shao et al., 2024a ; Wu et al., 2025b ) . In contrast, we propose to synthesize data that implicitly equips VLMs with an internal search mechanism—one that unfolds through a textual inner monologue, enabling the model to explore multiple potential solution paths: revisiting different image regions, verifying intermediate conclusions, identifying inconsistencies, and self-correcting when necessary. Our approach is complementary to prior methods and mirrors the behavior observed in reasoning models like R1 and commercial VLMs such as o1, where reasoning performance improves by scaling test-time inference. We emphasize we do not claim that long textual CoTs are inherently superior or the only way to scale test-time inference in VLMs. Rather, our goal is to synthesize data to equip models with such capability—an approach shown to be effective in SoTA reasoning models. Furthermore, given the difficulty of building reliable process verifiers for search in perceptual tasks, our data-centric method offers a practical alternative. Figure 1 : LongPerceptualThoughts is a new synthetic dataset with 30K long-thought traces for vision-centric tasks. Each trace contains diverse cognitive behaviors (e.g., verification, subgoal setting, and backtracking), akin to system-2 reasoning. CoTs generated by open-source VLMs often produce linear, rigid reasoning traces (top). In contrast, our novel data synthesis framework effectively expands these simple thoughts using frontier reasoning models, equipping VLMs with complex reasoning structures and rich cognitive behaviors—effectively distilling system-2 reasoning into instruction-tuned VLMs. In this work, we take a first step toward a scalable method that synthesizes long CoT data for vision-centric tasks. Specifically, we propose a novel three-stage data synthesis framework that: (1) generates synthetic verifiable multiple-choice questions from dense image captions, (2) extracts simple CoTs from VLMs for those questions, and (3) expands these simple CoTs into richer, long-form reasoning traces using frontier reasoning models. Notably, our framework performs three layers of synthesis: one to generate questions , one to think, and the last one to think harder. As shown in Fig. 1 , using our framework, we generate LongPerceptualThoughts , a dataset of 30 k examples for both supervised fine-tuning (SFT) and direct preference optimization (DPO), and use it to fine-tune a strong instruction-tuned VLM. The resulting model shows an average +3.4 points improvement across 5 vision-centric benchmarks, including a gain of +11.8 points on V ∗ Bench, while typical multimodal reasoning datasets fail to improve the base VLM due to overthinking. Notably, despite being tuned for vision tasks, it also improves on the challenging text reasoning benchmark MMLU-Pro by +2 points. 2 Synthesize Long CoT Data for Vision-Centric Tasks In this section, we introduce a novel data synthesis framework to synthesize long chain-of-thought (CoT) data for fine-tuning a vision-language model (VLM). Inspired by DeepSeek’s R1, we are interested in collecting data consisting of thoughts and answers in the format of <think> thought </think> <answer> answer </answer> . We start by discussing two desired properties of reasoning data for vision-centric tasks in Sec. 2.1 . Based on these two properties, we explain our data synthesis framework in Sec. 2.3 . Finally, we use the synthesized long CoT data to construct LongPerceptualThoughts, consisting of both SFT and preference datasets in Sec. 2.4 . 2.1 Desired Properties in Long Chain-of-Thought Inspired by the recent success in OpenAI’s o1 and DeepSeeks’s R1, we further define Long CoT as an extended, structured rationale that mirrors how a human might approach complex visual reasoning tasks. Unlike the short, linear responses typically produced by current open-source VLMs, Long CoTs explore alternative solutions, verifying intermediate steps, and adjusting course when necessary. Drawing on the framework proposed in Gandhi et al. (2025) , we characterize Long CoTs in vision-centric tasks through three core cognitive behaviors: verification (checking intermediate conclusions for correctness), backtracking (recognizing and revising failed solution paths), and subgoal setting (breaking down the task into smaller, solvable components). These cognitive behaviors have been observed in LLM to increase performance by scaling test-time compute ( Muennighoff et al., 2025 ) . To study cognitive behaviors in vision-centric tasks, we begin by analyzing the outputs of strong instruction-tuned VLMs, following Gandhi et al. (2025) . Despite its general capabilities, the model rarely displays the cognitive behaviors described earlier. The responses are often shallow and rigid, lacking the iterative, self-corrective reasoning we aim to capture. Figure 3(a) quantifies this gap between the response from Qwen2.5-VL-7B-Instruct and Gemini 2.0 Flash Thinking. At the end of this section, we introduce LongPerceptualThoughts that drastically diversifies the standard CoT with the desired cognitive behaviors.