Detached Skip-Links and R R -Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR

Ziye Yuan, Ruchang Yao, Chengxin Zheng, Yusheng Zhao, Daxiang Dong, Ming Zhang · 2026

Read Detached Skip-Links and R R -Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR by Ziye Yuan, Ruchang Yao, Chengxin Zheng, Yusheng Zhao, Daxiang Dong, Ming Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20020.txt.

Introduction

Yuan and co-authors fix a training flaw that hurts text reading in multimodal language models. These models reason well but often miss fine visual detail. The authors trace the problem to skip links in feature fusion. Those links let high level goals overwrite early visual signals through back propagation. Their detached skip links reuse shallow features forward but block gradients on that path. No new learnable parameters are added. A probe tool, R-Probe, checks if fine detail survives in the visual tokens.

Catalog record

CREATORZiye Yuan, Ruchang Yao, Chengxin Zheng, Yusheng Zhao, Daxiang Dong, Ming Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66316
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20020

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detached Skip-Links and R R -Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR Ziye Yuan, Ruchang Yao, Chengxin Zheng, Yusheng Zhao, Daxiang Dong, Ming Zhang Abstract Multimodal large language models (MLLMs) excel at high-level reasoning yet fail on OCR tasks where fine-grained visual details are compromised or misaligned. We identify an overlooked optimization issue in multi-layer feature fusion. Skip pathways introduce direct back-propagation paths from high-level semantic objectives to early visual layers. This mechanism overwrites low-level signals and destabilizes training. To mitigate this gradient interference, we propose Detached Skip-Links, a minimal modification that reuses shallow features in the forward pass while stopping gradients through the skip branch during joint training. This asymmetric design reduces gradient interference, improving stability and convergence without adding learnable parameters. To diagnose whether fine-grained information is preserved and usable by an LLM, we introduce R R -Probe, which measures pixel-level reconstructability of projected visual tokens using a shallow decoder initialized from the first quarter of the LLM layers. Across multiple ViT backbones and multimodal benchmarks, and at scales up to 7M training samples, our approach consistently improves OCR-centric benchmarks and delivers clear gains on general multimodal tasks. 1 Introduction Multimodal large language models (MLLMs) have rapidly advanced the integration of vision and language, showing strong performance in high-level semantic reasoning and dialogue. ( 37 ; 1 ; 42 ; 6 ) However, they still exhibit a clear performance gap on low-level perception tasks, particularly Optical Character Recognition (OCR) and fine-grained visual grounding. Existing benchmarks show that even state-of-the-art models often hallucinate text in dense documents or fail to resolve small objects in high-resolution scenes ( 18 ; 31 ; 23 ) . Previous studies identify the pre-trained Vision Transformer (ViT) as a primary bottleneck ( 29 ) . By design, contrastive objectives (e.g., CLIP) encourage semantic alignment, pushing the encoder to abstract away spatial details to align with global text descriptions ( 40 ; 8 ) . To mitigate this information loss, recent approaches either incorporate auxiliary objectives such as reconstruction losses ( 17 ; 41 ) , or rely on end-to-end generative supervision from the multimodal LLM through its next-token prediction (NTP) loss ( 20 ; 11 ) . In parallel, architectural designs increasingly adopt multi-layer fusion to incorporate shallow features that retain geometric and pixel-level information ( 46 ; 43 ; 28 ) . While intuitively promising, we identify a previously under-discussed optimization issue in this fusion-based architectures. Straightforward fusion establishes direct backpropagation paths from the LLM’s semantic objectives to early visual blocks. This subjects shallow layers originally optimized for low-level patterns to conflicting high-level supervision, resulting in gradient interference and training instability, as in Figure 2 . To address this trade-off between spatial detail and training stability, we propose Detached Skip-Links (Fig. 1 ). By stopping gradients on shallow features before fusion, we decouple feature aggregation from gradient optimization. This mechanism passes fine-grained visual details to the LLM while preventing semantic-heavy gradients from destabilizing shallow layers, without sacrificing the simplicity of skip-connections. Both theoretical analysis and empirical results show that this operation reduces gradient conflict, leading to improved training stability and faster convergence. These effects are consistent across different ViT backbones and remain stable when scaling training to 7M samples. Figure 1 : Overview of Detached Skip Links. Intermediate features are concatenated with the final output along the channel dimension, here S denotes stride. We apply a stop-gradient operation to shallow skip features before fusion. This design effectively delivers fine-grained details to the LLM while shielding early layers from optimization conflicts. Beyond optimization, a major challenge in improving fine-grained perception is the lack of a reliable diagnostic metric. Standard downstream benchmarks are often noisy proxies for visual capability, as MLLMs can bypass perception by exploiting language priors or parametric knowledge ( 21 ) . Existing probing methods typically use linear classifiers ( 34 ; 3 ; 16 ) to assess representation quality, but such tasks are insufficient for measuring fine-grained visual information. To systematically quantify the visual signal effectively transmitted to the LLM, we introduce the Reconstruction Probe ( R -Probe). Moving beyond traditional linear separability tests, R -Probe assesses the recoverability of visual details through pixel-level reconstruction. By initializing the reconstruction head with the first quarter layers of the target LLM, we simulate the actual information injection process, probing features exactly as they are ”perceived” by the language model. We posit that minimal reconstruction loss from the projected tokens serves as a proxy for effective information preservation. Our main contributions are summarized as follows: • Introduce Detached Skip-Links : reuse shallow ViT features for fusion while stopping gradients through the selected skip branch, reducing gradient interference and improving training stability without additional parameters. • Propose R -Probe : a reconstruction-based diagnostic that measures whether projected visual tokens retain fine-grained information and remain directly decodable by an LLM-initialized shallow decoder. • Demonstrate effectiveness at scale : extensive experiments across ViT backbones and 22 benchmarks show strong gains on OCR-centric tasks and consistent improvements on general multimodal evaluation. 2 Related Work OCR-Centric Multimodal Large Language Models. Recent advances have shifted from modular OCR pipelines to end-to-end MLLMs that internalize text recognition. While some systems ( 13 ) focused on localized extraction, recent large-scale MLLMs ( 10 ; 32 ; 45 ; 12 ; 38 ; 39 ; 6 ; 15 ; 42 ) demonstrate that massive pretraining with OCR-synthesized data yields strong character-level capabilities. To capture fine-grained details, modern architectures often employ dynamic resolution or image-splitting strategies ( 10 ; 6 ) . However, effectively injecting these high-fidelity features into the LLM context remains an open challenge. While earlier general-purpose models relied on heavy bridging modules like ( 4 ; 25 ; 20 ) , recent OCR-specific approaches favor multi-scale fusion ( 47 ; 26 ) or deep cross-attention ( 6 ; 9 ) . Representative examples include TextHawk ( 49 ) , with token compression and multi-level cross-attention, and VLM-FO1 ( 30 ) , which strengthens regional OCR using auxiliary high-resolution encoders and explicit region tokens. In contrast to complicated architectures, our work provides a lightweight training-time solution without introducing new modules, and is orthogonal to these architectural designs. Multi-layer Fusion Strategies. To mitigate information loss in deep ViTs, researchers have explored leveraging intermediate features. Methods like DeepStack ( 33 ) , DenseConnector ( 46 ) and ( 19 ; 28 ) aggregate representations from multiple depths, while others combine signals from distinct vision backbones ( 36 ; 24 ; 45 ) or use hierarchical schemes ( 50 ) . While conceptually sound, such heterogeneous fusion often exhibits unstable optimization dynamics in practice. In the broader optimization literature, controlling or blocking gradient flow has been explored as a strategy to improve training stability in settings such as recurrent networks and representation learning ( 5 ; 48 ) . A plausible source of instability is the interference between high-level semantic gradients from the LLM and the shallow layers’ role in preserving fine-grained visual details, such as character strokes. As a result, naive fusion can lead to gradient conflicts and impair training stability. We therefore investigate whether decoupling feature propagation from gradient flow can alleviate such interference, while retaining the benefits of shallow

Read the complete text