FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding
Amit Agarwal, Srikant Panda, Kulbhushan Pachuri · 2025
Read FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding by Amit Agarwal, Srikant Panda, Kulbhushan Pachuri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17330.txt.
Introduction
Amit Agarwal, Srikant Panda, and Kulbhushan Pachauri present FS-DAG, a small graph network that reads rich documents with only a few training samples. Their paper from twenty twenty-five targets visually rich document understanding, where layout and look carry meaning. Few-shot settings starve big models. FS-DAG keeps its parameter count low while staying modular.
Domain backbones and language or vision backbones plug into the frame as needed. The design shrugs off OCR errors, misspellings, and domain drift, all common in live use. Tests on information extraction show faster settling and higher scores than top rivals. The authors stress the wider trend: lean models that hold their own. Code is public. The work suits extraction jobs where compute is tight and forms keep changing.
Catalog record
| CREATOR | Amit Agarwal, Srikant Panda, Kulbhushan Pachuri |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding Amit Agarwal, Srikant Panda, Kulbhushan Pachuri Abstract In this work, we propose Few Shot Domain Adapting Graph (FS-DAG), a scalable and efficient model architecture for visually rich document understanding (VRDU) in few-shot settings. FS-DAG leverages domain-specific and language/vision specific backbones within a modular framework to adapt to diverse document types with minimal data. The model is robust to practical challenges such as handling OCR errors, misspellings, and domain shifts, which are critical in real-world deployments. FS-DAG is highly performant with less than 90M parameters, making it well-suited for complex real-world applications for Information Extraction (IE) tasks where computational resources are limited. We demonstrate FS-DAG’s capability through extensive experiments for information extraction task, showing significant improvements in convergence speed and performance compared to state-of-the-art methods. Additionally, this work highlights the ongoing progress in developing smaller, more efficient models that do not compromise on performance. 1 Introduction Recent advancements of Vision-Language Models (VLMs) ( Zhang et al. 2024 ; Pattnayak et al. 2024 ; Agarwal et al. 2024b ) , Large Multimodal Models (LMMs) ( Chen et al. 2024 ; Li et al. 2024 ) , and Large Language Models (LLMs) ( Brown 2020 ; Touvron et al. 2023 ) , have significantly enhanced performance across various natural language processing Pattnayak et al. 2025c and computer vision tasks. Despite their success, these models are often computationally expensive, requiring substantial resources that are impractical for many real-world industrial applications ( Sanh et al. 2019 ; Kaddour et al. 2023 ) . Furthermore, their ability to adapt to specific domains, especially in the context of visually rich documents (VRDs) remains limited due to the high cost of pre-training and fine-tuning on domain-specific data ( Li et al. 2021 ) . VRDs face challenges stemming from diverse layouts, domain-specific terminology, and text variations in style and size. OCR-free models tend to underperform compared to key-value models that utilize a separate OCR component, and even these models struggle with such variations. Large-scale models, with their monolithic architectures, often rely on vast data for domain adaptation, complicating their deployment. For example, state-of-the-art models like LayoutLM ( Xu et al. 2020a ) and its successors demand extensive fine-tuning for new domains, making their deployment both costly and time-consuming ( Huang et al. 2022 ) . To address these issues, we introduce FS-DAG, a few-shot learning framework designed for domain-specific document understanding with less than 90M parameters. Few-shot learning methods have gained attention for their ability to train models with limited labeled data, which is crucial in industrial applications where data scarcity is a common challenge. Our approach leverages a modular architecture that integrates domain-specific and language-specific feature extractors, allowing FS-DAG to adapt quickly to new domains with minimal data, thereby overcoming the barriers associated with large-scale models ( Lee et al. 2022 ) . Our approach emphasizes few-shot learning by leveraging Graph Neural Networks (GNNs) ( Khemani et al. 2024 ; Wu et al. 2020 ; Yin et al. 2024 ) to enable rapid adaptation, robustness to OCR errors, and reduced latency in real-world applications. We provide empirical evidence of the model’s performance through extensive experiments, showing significant improvements over larger methods with more than 100M parameters. To summarize, we make the following contributions to VRDU in a few-shot learning environment: 1. A modular framework for few-shot learning that efficiently combines domain-specific and language-specific textual and visual feature extractors in a graph-based architecture. 2. We propose shared positional embedding & consistent reading order for GNN along with various training strategies for the model’s robustness and effective adaptation with minimal data. 3. We provide comprehensive experimental results demonstrating that FS-DAG achieves state-of-the-art performance and robustness in few-shot learning scenarios while reducing latency and computational costs. 2 Related Work The development of efficient and scalable NLP models has gained significant attention in recent years, particularly with the rise of LLMs ( Pattnayak et al. ; Pattnayak et al. 2025a ; Patel et al. 2025 ) such as GPT-3 ( Brown 2020 ) , LlaMa ( Touvron et al. 2023 ) , Mixtrals ( Jiang et al. 2024 ) . While these models have achieved remarkable success in various tasks, their application in industrial settings remains challenging due to their high computational demands and difficulty in adapting to domain-specific tasks. Recent work have focused on enhancing the efficiency of these models through techniques such as model distillation ( Sanh et al. 2019 ) , pruning ( Cheng et al. 2024 ) , and efficient fine-tuning methods like LoRA ( Hu et al. 2022 ; Thomas et al. 2025 ) . These approaches aim to reduce the computational cost of LLMs while maintaining their performance, making them more suitable for deployment in resource-constrained environments. In the context of VRDU, graph-based models have shown promise, particularly in capturing the complex relationships between textual and visual elements in documents. Models such as SDMGR ( Sun et al. 2021 ) , DocParser ( Rausch et al. 2021 ) , PICK ( Yu et al. 2021 ) and others ( Liu et al. 2019 ; Rastogi et al. 2020 ; Yao et al. 2021 ) leverage GNNs to improve IE from documents. However, these models often require large amounts of training data and are not designed for quick adaptation to new domains. FS-DAG builds on these approaches by introducing a few-shot learning framework that can efficiently adapt to new document types with minimal data. This capability is particularly important in industrial applications, where labeled data is often limited, and the ability to quickly adapt to new domains is crucial. Additionally, FS-DAG addresses practical challenges such as robustness to OCR errors and domain shifts, which are common in real-world deployments. Figure 1: An illustration of the model architecture for FS-DAG. Given a document image (I); its text regions \{r_{i}\} are extracted using an OCR engine. We cluster and sort the \{r_{i}\} to create a reading sequence \{s\} ; textual features \{t_{i}\} are extracted using a linear projection layer on top of a pre-trained language model processing \{s\} . In contrast, visual features \{v_{i}\} are extracted using ROI-Align on top of the feature map from the Visual Model and \{r_{i}\} . The deep fusion module uses Kronecker product to fuse \{t_{i}\} and \{v_{i}\} to initialize the node features \{n_{i}\} . The node features are propagated and aggregated in the GNN during the message passing, which uses positional embedding \{p_{i}\} and multi-head attention to learn the edge features dynamically. The classification head will finally classify the node features into one of the key-value classes. 3 Our Approach Figure 1 illustrates our proposed model architecture. FS-DAG formulates the Key Information Extraction (KIE) ( Huang et al. 2019 ) task as a graph node classification problem using pre-trained feature extractors and graph multi-head attention in a few-shot learning environment. 3.1 Model Architecture The FS-DAG model Agarwal et al. 2024c is designed to address the unique challenges associated with VRDU in few-shot learning scenarios. Unlike traditional monolithic models ( Yu et al. 2021 ; Xu et al. 2020b ; Xu et al. 2020a ; Huang et al. 2022 ; Xu et al. 2021 ) that often require large amounts of