Internalized Reasoning for Long-Context Visual Document Understanding

Austin Veselka · 2026

Read Internalized Reasoning for Long-Context Visual Document Understanding by Austin Veselka (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02371.txt.

Introduction

Veselka builds a synthetic data pipeline for reasoning in long visual document understanding. The pipeline scores each page for question relevance, extracts textual evidence and orders it from most to least relevant. Supervised fine tuning on the resulting traces, gated by a control token, plus low strength model merging internalizes the reasoning skill. On Qwen3 VL the method scores 58 versus 57 for a far larger sibling model, and on Mistral it beats distillation by about 4 points while using about 12 fewer output tokens.

Catalog record

CREATORAustin Veselka
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43458
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.02371

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Internalized Reasoning for Long-Context Visual Document Understanding Austin Veselka Abstract Visual long-document understanding is critical for enterprise, legal, and scientific applications, yet the best performing open recipes have not explored reasoning, a capability which has driven leaps in math and code performance. We introduce a synthetic data pipeline for reasoning in long-document understanding that generates thinking traces by scoring each page for question relevance, extracting textual evidence and ordering it from most to least relevant. We apply SFT to the resulting traces within <think> tags, gated by a <cot> control token, and the resulting reasoning capability is internalized via low-strength model merging. We study Qwen3 VL 32B and Mistral Small 3.1 24B. With Qwen3 VL, we achieve 58.3 on MMLongBenchDoc, surpassing the 7 × \times larger Qwen3 VL 235B A22B (57.0). With Mistral, we show that synthetic reasoning outperforms distillation from the Thinking version’s traces by 3.8 points on MMLBD-C, and internalized reasoning exhibits 12.4 × \times fewer mean output tokens compared to explicit reasoning. We release our pipeline for reproducibility and further exploration 1 1 1 https://github.com/lightonai/distilabel/tree/lc_sft_pipelines . Built on a fork of an Apache 2.0 licensed package. . 1 Introduction Long-context (LC) visual language models (VLMs) for document understanding have natural applications in enterprise, legal, scientific and financial domains. The promise of powerful LC VLMs is an engine capable of performing a multitude of complex operations over documents in these domains: specific information retrieval, in-depth understanding for question-answering (QA), complex and multi-hop reasoning and QA, summarization and more. Reasoning models [ 36 ] have introduced test-time scaling as a new dimension of scaling for LLMs, with arbitrary length inference sequences removing bounds on the computable algorithms for transformers [ 50 , 32 , 25 ] and implying the potential to scale performance generally, including for the long-document operations above. While reasoning has proven extremely effective in domains with verifiable rewards, e.g. math and coding [ 36 , 9 , 37 ] , the lack of such high-signal rewards in QA limits most existing works to non-reasoning approaches. For instance, Docopilot [ 12 ] is an open VLM which builds a dataset from Arxiv, Sci-Hub and OpenReview and fine-tunes InternVL 2 2B and 8B with SFT. Recent work [ 43 ] has explored recipes for training long-document VLMs using CPT, SFT and LongPO, but does not explore methods for or benefits of reasoning over the document. The value of this domain has spurred the creation of challenging benchmarks such as MMLongBenchDoc (MMLBD) [ 31 ] which evaluate the model on documents up to hundreds of pages long with human written questions and answers. Only recently has any model surpassed GPT 4o [ 35 ] in performance (beginning with Qwen3 VL [ 42 ] ) and Qwen3 VL 235B A22B still falls well short of the human baseline, making it clear that this domain is still significantly under-explored. Additionally, Qwen3 VL 235B A22B Instruct and Thinking achieve equivalent performance, highlighting the discussed gap in leading models: frontier reasoning capabilities do not contribute to stronger document understanding. Motivated by this gap, we propose a synthetic data pipeline that generates structured reasoning traces via page-level evidence extraction, relevance scoring and bounded top- K sorting. We train Qwen3 VL 32B Instruct 2 2 2 Released under the Apache 2.0 license. (Qwen3 VL) and Mistral Small 3.1 24B 2 (Mistral) [ 33 ] and show that low-strength model merging via task arithmetic [ 20 ] makes the reasoning implicit. Further, there is a background of work related to internalized or implicit chain-of-thought reasoning. These methods include auxiliary losses over the answer rationale [ 18 , 51 ] , reasoning post answer [ 44 , 6 ] and hidden state distillation from thinking models [ 10 ] , but these works do not clarify whether the rationale improves training signal or modifies inference behavior. We believe our work is the first to show an implicit CoT that is an active inference-time capability rather than a training artifact alone. Contributions. • Synthetic reasoning pipeline. We present a synthetic reasoning pipeline for long-document VQA which achieves SOTA on MMLBD and MMLBD-C [ 43 ] with Qwen3 VL 32B Instruct, surpassing the teacher model Qwen3 VL 235B A22B with 7× fewer parameters. We further show that the method generalizes across model families with Mistral improving by +19% relative to the base model on both MMLBD and MMLBD-C. • Internalized reasoning. We show that reasoning can be internalized by low-strength model merging, exhibiting similar output lengths without thinking tokens to the no-think baseline (250 vs 140 for Qwen, 132 vs 160 for Mistral) relative to higher merge strength (Mistral \alpha=0.5 ) which uses thinking tokens in 77% of questions and averages 1637 output tokens. • Internalized reasoning is an inference-time capability. By gating reasoning behind a control token during training, we make it possible to study the effect of the reasoning during inference . We use this to show that our internalized reasoning is an inference-time capability with a significant impact on performance: removing the <cot> control token degrades performance on MMLBD by 3.8 points for Qwen and 2.1 points for Mistral. • Synthetic reasoning outperforms thinking traces. We find that our synthetic reasoning pipeline is more effective than traces from Qwen3 VL 235B A22B Thinking, indicating the strength of our trace design and highlighting the potential for future models to improve reasoning for long-document VQA. 2 Related work Long-context VLMs and training. Most prior work on long-context VLMs focuses on video processing [ 7 , 38 , 29 ] , while long-document understanding has been addressed primarily by closed models (GPT-4o [ 35 ] , Gemini [ 16 ] ) or open-weight models with underspecified training recipes [ 42 , 54 ] . Docopilot [ 12 ] fine-tunes InternVL2 on a curated 758K-sample document dataset and V2PE [ 15 ] extends VLM context to 1M tokens via variable position encoding. On the training methodology side, ProLong [ 14 ] establishes data recipes for efficient long-context language model training, while LongPO [ 5 ] and SoLoPO [ 39 ] use short-to-long preference optimization. Synthetic data generation for long-context training includes hard-negative construction [ 13 ] , bootstrapping from short-context models [ 46 ] , and realistic instruction synthesis [ 24 ] . In [ 43 ] , the authors provide open CPT, SFT, and LongPO recipes for document VLMs and achieve SOTA on MMLBD-C. None of the above works explore the benefits of reasoning over the document context. As such, we extend the recursive answer generation pipeline from [ 43 ] to the task of training reasoning models for long-document VQA. Reasoning models. Chain-of-thought prompting [ 49 ] demonstrated that step-by-step reasoning substantially improves LLM performance, spurring a paradigm shift toward trained reasoning models. OpenAI o1 [ 36 ] and DeepSeek-R1 [ 9 ] learn extended reasoning via reinforcement learning and thinking variants of Qwen3 VL [ 42 ] and GLM [ 54 ] extend this paradigm to multimodal models. LongFaith [ 52 ] generates faithful long-context reasoning traces with citation grounding and Temporal CoT [ 2 ] applies frame-level evidence selection for long videos—analogous to our page-level approach, but executes the pipeline during inference. Reasoning advances have been

Read the complete text