DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models

Xinrui Shi, Kai Liu, Ziqing Zhang, Jianze Li, Anqi Li, Yulun Zhang · 2026

Read DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models by Xinrui Shi, Kai Liu, Ziqing Zhang, Jianze Li, Anqi Li, Yulun Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26038.txt.

Introduction

Shi and five colleagues shore up small vision-language models for crowded scenes, where many objects, traits and links must be grounded and reasoned over in steps. Light models do well on plain benchmarks yet fail in dense scenes, partly because training never ties reasoning steps to visual items. The authors first build DRBench, with 14,573 questions over 2,943 images in five task types across three reasoning layers. Their DRScaffold training scheme then splits supervision into grounded parts. Results lift dense-scene reasoning while keeping the model light.

Catalog record

CREATORXinrui Shi, Kai Liu, Ziqing Zhang, Jianze Li, Anqi Li, Yulun Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60791
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26038

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models Xinrui Shi, Kai Liu, Ziqing Zhang, Jianze Li, Anqi Li, Yulun Zhang Abstract Lightweight vision-language models perform competitively on standard benchmarks yet fail systematically in dense-scene reasoning, where multiple objects, attributes, and relations must be jointly grounded and resolved through multi-step inference. Such capability is critical for real-world applications where models must reliably interpret cluttered environments. Yet existing training signals provide no explicit grounding between reasoning steps and the underlying visual entities and relations, leaving lightweight models free to generate fluent but visually unanchored reasoning chains. To address this gap, we first introduce DRBench , a benchmark of 14,573 questions across 2,943 images, organized into five task categories spanning three progressive reasoning layers. Building on DRBench, we propose DRScaffold , a supervised fine-tuning framework that decomposes the supervision target into four causally ordered stages, enforcing grounded reasoning without architectural modification. Experiments on three lightweight VLMs demonstrate substantial gains on DRBench while preserving or improving performance on general-purpose benchmarks. Notably, Qwen2.5-VL-3B trained with DRScaffold surpasses the frozen Qwen2.5-VL-32B on DRBench, demonstrating that structured supervision can substitute for a significant portion of model scale in dense-scene reasoning. Our code and models are available at https://github.com/irene-shi/DRScaffold . Figure 1 : Overview of our DRScaffold on an example in our DRBench. The base lightweight VLM model fails in dense-reasoning scenes. DRScaffold trains the model to ground key entities, build a relation graph, and reason over the grounded structures to produce the correct answer. 1 Introduction While large-scale Vision-Language Models (VLMs), such as GPT-4o 2 and Gemini 2.5 Pro 8 , have shown impressive capabilities, high computational cost and latency limit their use in real-world deployment. Practical applications ranging from domestic robots to autonomous vehicles require lightweight VLMs that operate under strict resource and latency constraints. This has motivated a growing line of work on mobile-scale VLMs, including state-of-the-art architectures, such as Qwen2.5-VL 3 and Phi4-multimodal 1 . Beyond general-purpose models, specialized systems have emerged, such as LINGO-2 34 and EM-VLM4AD 12 , which are designed for autonomous driving tasks. Further efficiency is achieved by MobileVLM-V2 7 , with only 1.1B parameters. Figure 2 : Qwen2.5-VL-3B vs 72B. However, these advances share a common blind spot: dense-scene reasoning. In such settings, multiple objects, attributes, and relations are tightly coupled, requiring multi-step reasoning over visual dependencies, which are ubiquitous in autonomous driving, robotic manipulation, and surveillance. As shown in Figure 1 , lightweight models fail on such queries, exhibiting relation misinterpretation, attribute misbinding, and hallucinated dependencies . Empirically, Figure 2 shows that the gap between lightweight and large VLMs is modest on standard benchmarks, but widens on dense-scene reasoning, indicating that standard benchmarks underestimate performance disparities in complex scenes. Given the strong reasoning ability of large VLMs, a practical solution is to leverage their outputs as supervision drafts to train light models. Yet current strategies remain insufficient due to a fundamental mismatch between the multi-stage reasoning process that dense-scene understanding requires and the undifferentiated form of existing training signals. Outcome-based supervision trains on final answers alone, discarding all intermediate reasoning 22 ; 21 . Chain-of-Thought supervision improves on this by distilling the teacher’s reasoning process, but treats it as a flat token sequence without structural constraints 11 ; 26 . Neither approach enforces alignment between reasoning steps and the underlying visual evidence. We argue that dense-scene reasoning requires four distinct stages that must be explicitly modeled: (1) entity grounding to identify and locate relevant objects, (2) relation modeling to resolve the spatial or logical dependencies among them, (3) stepwise reasoning to perform multi-step inference based on the structured relations, and (4) answer generation . This decomposition is not reflected in any existing training. Existing methods entangle reasoning stages into a monolithic sequence, which allows lightweight models to minimize training loss by mimicking fluent language patterns without verifying which visual entity each step refers to. This leads to reasoning that sounds plausible but is factually ungrounded, a failure mode that flat supervision cannot diagnose or prevent. To bridge this gap, we propose DRScaffold , a supervised fine-tuning framework that enhances D ense-scene R easoning through “ Scaffold ing”, enabling the model to learn complex structure layer by layer. First, we reformulate the supervision target as a schema-constrained pipeline with fixed slot order: detected entities, scene graph, step-wise reasoning, and final answer, so that the answer token is causally conditioned on previously grounded entities and verified relations. Second, training proceeds via staged gradient masking: each of the four training phases unlocks one additional field for loss computation, compelling the student to internalize each stage before proceeding to the next. Training and evaluating such a framework requires supervision that existing benchmarks do not provide. We introduce DRBench , a D ense-scene R easoning Bench mark of 14,573 questions across 2,943 images, organized into five categories in three reasoning layers. Beyond final correct answers, each sample contains structured reasoning process annotations, including key entities, scene graphs, and reasoning chains. These serve as supervision targets for DRScaffold. DRBench includes a False Premise Rejection category in which visually absent referents are embedded inside attribute or relation queries, providing a more challenging test than binary-polling benchmarks 19 ; 33 ; 14 ; 4 . Our contributions are summarized as follows: • We introduce DRBench, a dense-scene reasoning benchmark with five task categories across three progressive reasoning layers, providing structured intermediate reasoning annotations alongside a False Premise Rejection category for stricter hallucination evaluation. • We propose DRScaffold, a fine-tuning framework that decomposes the supervision target into a causally ordered four-stage schema and applies staged gradient masking to enforce structured reasoning in lightweight VLMs, requiring no architectural modifications. • Experiments on lightweight VLMs at 2-6B scale demonstrate that DRScaffold yields substantial gains on DRBench while preserving performance on established benchmarks. Qwen2.5-VL-3B trained with DRScaffold surpasses Qwen2.5-VL-32B on DRBench. 2 Related Work Figure 3 : Example DRBench annotation, with an image, question, answer, and structured intermediate annotations including key entities, a scene graph, and a staged reasoning chain. 2.1 Supervised Fine-Tuning Leveraging large models to generate supervision signals for training smaller models has become a scalable paradigm for reasoning-intensive tasks 31 ; 28 ; 22 . Unlike traditional knowledge distillation that aligns output distributions or intermediate feature representations 15 ; 30 ; 27 , recent approaches use large models to produce rich offline supervision, including instruction-following data 31 ; 28 , reasoning traces 17 ; 11 ; 16 , and preference data 32 . Among these, Chain-of-Thought (CoT) distillation has emerged as a particularly promising direction for reasoning-intensive tasks. Rather than training on final answers alone 22 ; 21 , the student model is trained to reproduce the teacher’s step-by-step reasoning process 17 ; 26 . DDCoT 38 separates multimodal reasoning into visual recognition and language reasoning, assigning each component to specialized models and integrating them into a joint CoT rationale. LLaVA-CoT 36 constructs a structured four-stage reasoning pipeline: summarization, visual interpretation, logical reasoning, and conclusion generation, demonstrating that structured stage tags significantly improve over untagged flat CoT. However, both DDCoT and LLaVA-CoT treat reasoning as a sequence of language stages without grounding each step to specific visual entities or relations. In dense-scene settings, this is where

Read the complete text