ROAP: A Reading-Order and Attention-Prior Pipeline for Optimizing Layout Transformers in Key Information Extraction
Tingwei Xie, Jinxing He, Yonghong Song · 2026
Read ROAP: A Reading-Order and Attention-Prior Pipeline for Optimizing Layout Transformers in Key Information Extraction by Tingwei Xie, Jinxing He, Yonghong Song (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05470.txt.
Introduction
Tingwei Xie and 2 co-authors present ROAP, a 2026 pipeline that helps layout transformers read documents in the right order. These models extract key information from forms and receipts, but they often ignore logical reading order. Visual tokens can also dilute attention away from the text that matters.
ROAP first builds reading sequences from complex layouts with an adaptive gap-based tree method. Those sequences enter attention as a position bias, so order guides the model directly. A second prior boosts text-to-text attention and suppresses visual noise. Tests on the FUNSD and CORD benchmarks improved both LayoutLMv3 and GeoLayoutLM backbones, without retraining them, supporting the design as a general add-on.
Catalog record
| CREATOR | Tingwei Xie, Jinxing He, Yonghong Song |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05470
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ROAP: A Reading-Order and Attention-Prior Pipeline for Optimizing Layout Transformers in Key Information Extraction Tingwei Xie, Jinxing He, Yonghong Song Abstract The efficacy of Multimodal Transformers in visually-rich document understanding (VrDU) is critically constrained by two inherent limitations: the lack of explicit modeling for logical reading order and the interference of visual tokens that dilutes attention on textual semantics. To address these challenges, this paper presents ROAP, a lightweight and architecture-agnostic pipeline designed to optimize attention distributions in Layout Transformers without altering their pre-trained backbones. The proposed pipeline first employs an Adaptive-XY-Gap (AXG-Tree) to robustly extract hierarchical reading sequences from complex layouts. These sequences are then integrated into the attention mechanism via a Reading-Order-Aware Relative Position Bias (RO-RPB). Furthermore, a Textual-Token Sub-block Attention Prior (TT-Prior) is introduced to adaptively suppress visual noise and enhance fine-grained text-text interactions. Extensive experiments on the FUNSD and CORD benchmarks demonstrate that ROAP consistently improves the performance of representative backbones, including LayoutLMv3 and GeoLayoutLM. These findings confirm that explicitly modeling reading logic and regulating modality interference are critical for robust document understanding, offering a scalable solution for complex layout analysis. The implementation code will be released at https://github.com/KevinYuLei/ROAP . I Introduction Visually-rich Document Understanding (VrDU) has witnessed a paradigm shift with the advent of multimodal Transformers, which jointly model textual, layout, and visual information to interpret complex documents such as forms, receipts, and invoices. State-of-the-art models, such as LayoutLMv3 [ 6 ] and GeoLayoutLM [ 11 ] , typically rely on 2D spatial embeddings to capture the geometric arrangement of tokens. However, spatial proximity in a 2D plane does not always translate to semantic continuity. In scenarios involving multi-column layouts, rotated text, or dense tables, the lack of an explicit logical reading order (RO) often causes models to misinterpret the sequential flow of information, leading to errors in downstream tasks that require rigorous semantic reasoning. Furthermore, the integration of visual modalities introduces a secondary challenge: attention interference. While visual tokens (e.g., image patches) provide essential contextual cues, they can disproportionately dominate the self-attention mechanism due to their high spatial salience and quantity. This phenomenon, often referred to as modality competition, dilutes the model’s focus on textual semantics, particularly within the textual-token interaction sub-blocks. Standard attention mechanisms lack structural constraints to mitigate this noise, making it difficult for the model to maintain robust context modeling amidst irrelevant visual signals. To address these limitations, we propose a novel pipeline that harmonizes logical reading order with robust attention mechanisms. We first introduce the Adaptive-XY-Gap Tree (AXG-Tree), a data-driven algorithm that recursively parses document layouts to generate hierarchically consistent reading sequences, robust to skew and irregular spacing. Building upon this, we present ROAP , a unified optimization pipeline designed to inject this logical order into the Transformer architecture. ROAP is a lightweight and architecture-agnostic optimization pipeline that augments layout-aware Transformers by explicitly encoding AXG-Tree-derived sequences via R eading- O rder-Aware Relative Position Biases (RO-RPB) and utilizing a Textual-Token sub-block A ttention P rior (TT-Prior) to filter visual noise. Crucially, it can be seamlessly integrated into any self-attention-based document understanding model, including LayoutLMv3 and GeoLayoutLM, without altering their original architectures. Our contributions are summarized as follows: (1) We propose the Adaptive-XY-Gap Tree (AXG-Tree) for robust reading order extraction in complex layouts. (2) We introduce the Reading-Order-Aware Relative Position Bias (RO-RPB) to explicitly encode logical sequences into the self-attention mechanism without disrupting the original spatial perception. (3) We design the Textual-Token Sub-block Attention Prior (TT-Prior) to adaptively suppress visual modality noise and refine fine-grained text-text interactions. (4) Extensive experiments on FUNSD and CORD benchmarks demonstrate that ROAP consistently improves performance, achieving significant gains on the Semantic Entity Recognition (SER) and Relation Extraction (RE) tasks. II Related Work II-A Reading Order Modeling in Document Understanding Reading order (RO) information plays a pivotal role in VrDU, as it determines how textual and layout cues are sequentially perceived to form coherent semantics. Early layout-aware models often employed heuristic algorithms such as XY-Cut [ 3 ] to infer reading sequences from the geometric arrangements of text boxes. Although these rule-based methods demonstrate the feasibility of deriving RO directly from spatial layouts, they are sensitive to irregular alignments, skewed orientations, and multi-column structures, which greatly limit their robustness across diverse document formats. Fig. 1: Overall architecture of the proposed ROAP pipline. Subsequent studies have highlighted that accurate RO modeling can substantially enhance downstream performance. [ 22 ] proposed the Token Path Prediction (TPP) framework to address disordered input issues in OCR-based document understanding. By predicting entity mentions as token paths rather than sequential labels, TPP effectively mitigates the dependence on strictly ordered token sequences. However, this approach introduces a dense graph structure among tokens, leading to increased computational complexity and memory cost, and its reliance on grid-level prediction makes integration into pre-trained multimodal frameworks less straightforward. Similarly, [ 14 ] incorporated layout-aware reading order knowledge into the pre-training objectives of multimodal transformers, achieving performance gains on form and receipt understanding tasks. Nevertheless, this method demands extensive labeled data and high computational resources during pre-training, restricting its scalability and flexibility in fine-tuning stages. Recent work such as [ 23 ] further formalized reading order prediction as an ordering-relation extraction problem, where models learn pairwise order relations among text entities to reconstruct logical reading paths. Although conceptually elegant, these relation-based methods often require detailed supervision and complex annotation schemes, which hinder practical deployment in large-scale scenarios. Based on the above observations, this paper proposes the AXG-Tree, which efficiently infers robust reading-order indices and naturally integrates them into transformer attention through relative positional bias encoding. II-B Attention Interference Between Textual and Visual Tokens Transformer-based architectures have become central to VrDU, yet their self-attention mechanism is easily disrupted by the heterogeneous composition of multimodal token sequences. In real scanned documents, textual tokens and visual/layout tokens coexist, and prior studies have shown that these modalities often compete for attention capacity. [ 1 ] highlight that visual embeddings—though similar in number to textual tokens—can draw substantial attention mass due to their high spatial salience, thereby weakening the model’s ability to preserve fine-grained linguistic relationships. Similar interference phenomena have been observed in other multimodal Transformer systems. [ 12 ] show that treating all tokens uniformly allows visually dominant but semantically weak regions to attract disproportionate attention, introducing noise into the reasoning process. From an efficiency perspective, [ 21 ] demonstrate that long multimodal sequences amplify such interference, as visual tokens accumulate rapidly with increasing image resolution. Likewise, [ 24 ] report that excessive visual tokens complicate attention routing and degrade semantic consistency, particularly in forms and receipts where fine textual distinctions are essential. Beyond efficiency considerations, [ 16 ] provide mechanistic evidence that Transformer attention is highly sensitive to token heterogeneity, further confirming that non-textual tokens can distort attention patterns even without dominating the sequence length. Collectively, these findings reveal that visual tokens, by virtue of their spatial prominence, often compete with textual tokens for attention and obscure meaningful linguistic interactions. This interference is most detrimental within the textual-token (TT) sub-block of the attention matrix, where accurate modeling of intra-text relations is crucial for downstream VrDU tasks. Motivated by these observations, we introduce a TT-Prior mechanism that injects structural priors into the TT