Qianfan-OCR: A Unified End-to-End Model for Document Intelligence
Baidu Qianfan Team · 2026
Read Qianfan-OCR: A Unified End-to-End Model for Document Intelligence by Baidu Qianfan Team (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13398.txt.
Introduction
Daxiang Dong and co-authors present Qianfan-OCR, an end-to-end model for document intelligence. The 4B-parameter vision-language model unifies document parsing, layout analysis, and document understanding in one design. It converts images to Markdown and handles table extraction, chart questions, and key information extraction by prompt. A Layout-as-Thought phase first writes bounding boxes, element types, and reading order before final output. That step restores layout grounding and lifts accuracy on complex layouts. It ranked first among end-to-end models on two benchmarks.
Catalog record
| CREATOR | Baidu Qianfan Team |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54853 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13398
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Qianfan-OCR: A Unified End-to-End Model for Document Intelligence Baidu Qianfan Team Abstract We present Qianfan-OCR, a 4B-parameter end-to-end document intelligence model that unifies document parsing, layout analysis, and document understanding within a single vision-language architecture. Unlike traditional multi-stage OCR pipelines that chain separate layout detection, text recognition, and language comprehension modules, Qianfan-OCR performs direct image-to-Markdown conversion and supports a broad range of prompt-driven tasks – from structured document parsing and table extraction to chart understanding, document question answering, and key information extraction – all within one model. A practical limitation of end-to-end OCR is the loss of explicit layout analysis, a capability that pipeline users routinely rely on for element localization and type classification. We introduce Layout-as-Thought to bridge this gap: an optional thinking phase triggered by ⟨ \langle think ⟩ \rangle tokens, where the model generates structured layout representations (bounding boxes, element types, and reading order) before producing final outputs. This mechanism serves two purposes: (1) it recovers layout analysis functionality within the end-to-end paradigm, enabling users to obtain spatial grounding results directly; and (2) it provides targeted accuracy improvements on documents with complex layouts, cluttered elements, or non-standard reading orders, where structural priors help resolve recognition ambiguities. On OCR-specific benchmarks, Qianfan-OCR ranks first among all end-to-end models on OmniDocBench v1.5 (93.12) and OlmOCR Bench (79.8). It also achieves strong results on general OCR benchmarks including OCRBench (880), OCRBenchv2, and CCOCR, as well as document understanding tasks such as DocVQA, ChartQA, and CharXiv, matching general vision-language models of comparable scale. On public Key Information Extraction benchmarks, Qianfan-OCR achieves the highest average score, surpassing Gemini-3.1-Pro, Gemini-3-Pro, Seed-2.0, and Qwen3-VL-235B-A22B. The model is publicly accessible through Baidu AI Cloud Qianfan platform, with usage examples and best practices available at https://github.com/baidubce/Qianfan-VL . \setCJKmainfont [BoldFont=FandolSong-Bold.otf]FandolSong-Regular.otf \setCJKsansfont [BoldFont=FandolHei-Bold.otf]FandolHei-Regular.otf \setCJKmonofont FandolFang-Regular.otf 1 Introduction Current OCR systems face a three-way trade-off between cost, accuracy, and capability. Traditional OCR pipelines based on small specialized models offer low inference cost and high throughput, but require complex multi-stage preprocessing and postprocessing to handle diverse document layouts. Specialized OCR large models [ 22 , 23 , 10 , 18 ] improve accuracy through two-stage architectures – layout detection followed by element-wise recognition – but introduce deployment complexity, inter-stage error propagation, and irreversible loss of visual context during text extraction. General vision-language models [ 13 , 7 ] offer broad multimodal capabilities but incur higher inference costs and underperform specialized systems on structured document parsing tasks requiring precise layout preservation. In practice, industrial OCR applications – document retrieval with chunking and indexing, contract review, key information extraction from receipts and certificates – often chain detection models, OCR models, and separate LLMs for downstream understanding. This fragmented approach increases deployment cost, limits end-to-end optimization, and requires careful orchestration of heterogeneous components. We introduce Qianfan-OCR, a 4B-parameter unified end-to-end model that addresses these limitations with three key designs: End-to-End Architecture: Qianfan-OCR integrates layout analysis, text recognition, and semantic understanding into a single vision-language model, eliminating inter-stage error propagation and enabling joint optimization across all tasks. The end-to-end design allows the model to retain full visual context throughout processing – spatial relationships, chart structures, and formatting that pipeline systems discard during text extraction. For tasks that do not require explicit layout analysis (e.g., simple document transcription or scene text recognition), the model directly outputs results without mandatory layout preprocessing. Layout-as-Thought: A practical limitation of end-to-end OCR is the loss of explicit layout analysis – a capability that pipeline systems inherently provide through dedicated detection modules. Layout-as-Thought recovers this within the end-to-end paradigm: an optional thinking phase triggered by \langle think \rangle tokens, where the model generates bounding boxes, element types, and reading order before producing final outputs. This serves two purposes: (1) functional – users obtain structured layout results (element localization, type classification, spatial grounding) directly from an end-to-end model, bridging a key functionality gap relative to pipeline systems; (2) enhancement – the explicit structural priors help resolve ambiguities in documents with cluttered elements, complex multi-column layouts, or non-standard reading orders. For well-structured documents where the model already performs well, the layout phase is unnecessary; it targets the subset of challenging cases where structural reasoning provides measurable gains. Unified OCR and Understanding: Beyond conventional OCR tasks (document parsing, handwriting recognition, table extraction), Qianfan-OCR handles cognitively demanding tasks including chart understanding, document question answering, and key information extraction – tasks requiring both precise text perception and semantic reasoning. Traditional OCR models lack comprehension capabilities, limiting them to character-level extraction; general VLMs possess reasoning abilities but underperform on structured parsing. Qianfan-OCR bridges this divide, combining OCR-specialist-level accuracy with document understanding capabilities in a single model controllable through prompts. Figure 1: Performance on OmniDocBench v1.5 across pipeline (left) and end-to-end (right) models. Qianfan-OCR (red) achieves 93.12, ranking first among all end-to-end models. The red dashed line indicates Qianfan-OCR’s score for cross-category comparison. Figure 2: Performance on general OCR and document understanding benchmarks. Top row: OlmOCR Bench, OCRBench, OCRBenchv2 (en), and OCRBenchv2 (zh). Bottom row: CCOCR-multilan, and document understanding tasks (DocVQA, TextVQA, OCRVQA) where two-stage OCR+LLM pipelines (hatched bars) show significant degradation compared to end-to-end models. 2 Related Work We review three technical routes in OCR and position Qianfan-OCR relative to each. Pipeline OCR Systems. Pipeline systems 9 decompose document parsing into layout detection, element-wise recognition, and rule-based assembly. Recent systems such as PaddleOCR-VL 10 , MonkeyOCR, and MinerU 2.5 pair lightweight detection models with VLM-based recognizers, achieving strong accuracy with modular efficiency. Their key advantage is explicit layout analysis output (bounding boxes, element types), but they suffer from inter-stage error propagation and irreversible loss of visual context during text extraction. Qianfan-OCR recovers the layout analysis capability through Layout-as-Thought while avoiding the pipeline’s cascading error problem. End-to-End OCR Models. End-to-end approaches directly map document images to structured outputs. Nougat 5 demonstrated feasibility on academic papers; GOT-OCR 2.0 22 broadened format support (Markdown, LaTeX, TikZ) at 580M parameters; DeepSeek-OCR 23 introduced context optical compression for efficiency; olmOCR 18 scaled SFT-based training on large-scale web documents, while its successor olmOCR 2 further introduced GRPO reinforcement learning with unit-test rewards. More recently, Dolphin v2 proposed analyze-then-parse, and Logics-Parsing and Infinity-Parser explored layout-aware RL for structure prediction. These models primarily focus on recognition accuracy or efficiency but lack explicit layout analysis output – a functionality gap that Qianfan-OCR’s Layout-as-Thought addresses. Qianfan-OCR relies on supervised fine-tuning with high-quality layout annotations, a complementary paradigm that future work could augment with reinforcement learning. General Vision-Language Models. Large VLMs such as Qwen-VL 3 , 4 , InternVL 7 , 26 , and Gemini exhibit OCR capabilities as a byproduct of broad multimodal training, but are not optimized for structured document parsing: they incur higher inference costs, lack fine-grained layout control, and underperform specialized systems on structure-sensitive metrics (e.g., table TEDS, reading order accuracy). Qianfan-OCR targets OCR-specialist-level accuracy at comparable inference cost to these models, while additionally supporting explicit layout analysis and prompt-driven task control. Figure 3: Architectural comparison between traditional two-stage OCR pipeline and Qianfan-OCR’s end-to-end approach. (a) Traditional pipeline systems separate layout analysis and content recognition into independent stages, suffering from error propagation and irreversible loss of visual context. (b) Qianfan-OCR unifies all processing into a single vision-language model, accepting custom prompts for flexible task