Extract Information from Hybrid Long Documents Leveraging LLMs: A Framework and Dataset

Chongjian Yue 2, * , Xinrun Xu 3, * , Xiaojun Ma 1, † , Lun Du 4, † Zhiming Ding 3 , Shi Han 1 , Dongmei Zhang 1 , Qi Zhang 1 · 2024

Read Extract Information from Hybrid Long Documents Leveraging LLMs: A Framework and Dataset by Chongjian Yue 2, * , Xinrun Xu 3, * , Xiaojun Ma 1, † , Lun Du 4, † Zhiming Ding 3 , Shi Han 1 , Dongmei Zhang 1 , Qi Zhang 1 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20072.txt.

Introduction

Chongjian Yue and seven co-authors build a framework for pulling facts out of long mixed documents in a 2024 paper. Such documents blend prose and tables, and they run far past the length a language model can read at once.

Their framework selects and sums up the useful parts of a document before the model works on them. Tests showed that a simple way of writing out tables is enough for models to grasp them. The basic setup also coped with many hard cases, and careful prompt design lifted results further. To help later work, the team released a data set drawn from financial reports. It asks models to extract numbers, a common real-world need.

Catalog record

CREATORChongjian Yue 2, * , Xinrun Xu 3, * , Xiaojun Ma 1, † , Lun Du 4, † Zhiming Ding 3 , Shi Han 1 , Dongmei Zhang 1 , Qi Zhang 1
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20072

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Extract Information from Hybrid Long Documents Leveraging LLMs: A Framework and Dataset Chongjian Yue 2, * , Xinrun Xu 3, * , Xiaojun Ma 1, † , Lun Du 4, † Zhiming Ding 3 , Shi Han 1 , Dongmei Zhang 1 , Qi Zhang 1 Abstract Large Language Models (LLMs) demonstrate exceptional performance in textual understanding and tabular reasoning tasks. However, their ability to comprehend and analyze hybrid text, containing textual and tabular data, remains unexplored. The hybrid text often appears in the form of hybrid long documents (HLDs), which far exceed the token limit of LLMs. Consequently, we apply an A utomated I nformation E xtraction framework ( AIE ) to enable LLMs to process the HLDs and carry out experiments to analyse four important aspects of information extraction from HLDs. Given the findings: 1) The effective way to select and summarize the useful part of a HLD. 2) An easy table serialization way is enough for LLMs to understand tables. 3) The naive AIE has adaptability in many complex scenarios. 4) The useful prompt engineering to enhance LLMs on HLDs. To address the issue of dataset scarcity in HLDs and support future work, we also propose the Fi nancial Reports N umerical E xtraction ( FINE ) dataset. The dataset and code are publicly available in the attachments. Email: [email protected], [email protected], [email protected], [email protected] [email protected], {shihan, dongmeiz, zhang.qi}@microsoft.com 1 1 footnotetext: Co-1st Authors and work done as interns at MSRA. 2 2 footnotetext: Corresponding Authors. I Introduction Large Language Models (LLMs) have demonstrated exceptional capabilities in understanding, analyzing and reasoning textual and tabular data independently, as evidenced by studies like [ 1 , 2 , 3 , 4 ] . However, their application to hybrid long documents (HLDs), which intricately weave together textual and tabular content, remains relatively unexplored [ 5 ] . This work addresses this gap by investigating the potential of LLMs for information extraction from HLDs, introducing an Automated Information Extraction framework called AIE. Given the length constraints of LLMs, directly processing entire HLDs is impractical and simple truncation leads to significant information loss [ 6 , 7 , 8 , 9 ] . AIE tackles this challenge by splitting HLDs into manageable segments and leveraging LLMs to extract relevant information from these segments. Our research delves into four key challenges associated with HLD information extraction: C1. Effective Selection and Summarization of Relevant Segments: With keyword-related information scattered across segments, effectively identifying and summarizing relevant content is crucial. We compare the “Refine” and “Map-Reduce” summarization strategies, exploring the trade-off between accuracy and efficiency. We also investigate the impact of varying the number of retrieved segments based on keyword similarity. C2. Optimal Table Serialization for LLMs: Tables are integral to HLDs, but LLMs can’t directly interpret tabular data. We explore different table serialization formats to identify the optimal representation for LLM comprehension. Our findings indicate that a simple, less hierarchical format is sufficient for LLMs to effectively understand tabular information. C3. Adaptability of AIE: HLDs encompass a diverse range of domains and complexities. We evaluate AIE’s adaptability through experiments assessing its performance across various domains, its handling of ambiguous expressions, and its compatibility with LLMs possessing different capabilities. C4. Prompt Engineering for Enhanced Information Extraction: Prompt engineering significantly influences LLM performance. We investigate and present effective prompt engineering techniques tailored for AIE in HLD information extraction. These include strategies for numerical precision enhancement, keyword completion leveraging document metadata, and effective few-shot learning approaches. The subsequent sections detail the AIE framework, incorporating globally optimal settings determined through our analysis (Section II ). We introduce the experimental dataset and evaluation metrics (Section III ) and present a comprehensive performance evaluation of AIE. Finally, we delve into a detailed analysis of the impact of various design choices on AIE’s effectiveness. II AIE Framework Fig. 1 : The AIE framework illustrates the end-to-end IE process, consisting of four modules: Segmentation, Retrieval, Summarization, and Extraction, extracting the keyword-corresponding value from the summary. To enable LLMs to process HLDs, we propose the Automated Information Extraction (AIE) framework, consisting of four modules: Segmentation, Retrieval, Summarization, and Extraction (Figure 1 ). AIE segments documents, retrieves keyword-relevant segments based on embedding similarity, summarizes these segments and finally extracts the target value. The Segmentation Module splits HLDs into manageable segments for LLMs. It involves Serialization , converting tables into text using the simple yet effective PLAIN method; Split , dividing overlong elements into smaller sub-elements to avoid information loss; and Merge , concatenating adjacent small elements to maintain semantic relationships. The Retrieval Module employs embedding-based retrieval [ 10 ] to avoid processing all segments [ 11 , 12 ] . It calculates the similarity between each segment and the keyword using the Sentence-Transformer model [ 13 ] and retrieves the top-ranked segments. The Summarization Module uses LLMs to generate a concise summary from the retrieved segments, capturing relevant information related to the keyword. We employ the Refine Strategy , which iteratively updates an evolving summary with information from each segment. Finally, the Extraction Module extracts the precise numerical value from the generated summary using LLMs and a tailored Extraction Prompt . We utilize three key prompt engineering techniques to enhance AIE’s performance: Numerical Precision Enhancement ensures accurate numerical extraction, crucial for financial analysis, using Direct and Shot-Precision methods; Keyword Completion improves IE accuracy by completing incomplete user-provided keywords using document metadata; and Few-Shot Learning guides LLMs to understand the task effectively through a single, well-designed shot. III Dataset and Evaluation Metrics TABLE I : Basic statistics for datasets. Dataset FINE WIKIR MPP Max # tokens 234,900 58,512 123,105 Min # tokens 13,022 13,548 3,672 Avg. # tokens 59,464.3 30,922.1 17,553.05 To evaluate LLMs’ capacity for HLD comprehension, we conduct experiments using three datasets: FINE : A new dataset with financial KPIs extracted from SEC’s EDGAR (Table I ). WIKIR [ 14 ] : Extracts key-value pairs from Wikipedia pages and Wikidata. MPP [ 15 ] : Extracts chemical material properties from scientific papers. FINE utilizes the Relative Error Tolerance Accuracy (RETA) metric due to varying numerical precision in financial reports. RETA considers predictions correct if their relative error is within a specified threshold (e.g., RETA X% means predictions with a relative error of no more than X% are considered correct) By setting different RETA levels, we can assess the model’s performance according to various practical requirements and gain a comprehensive understanding of its capabilities in IE from financial reports. WIKIR and MPP use Accuracy (Acc) as their ground truth values don’t exhibit precision variations. IV Experiment (a) Comparison using GPT-3.5. (b) Comparison using GPT-4. Fig. 2 : Comparison of the Naive method and AIE at different RETA levels on FINE. Fig. 3 : Comparison of the Naive method and AIE on WIKIR and MPP using GPT-3.5. We compare our proposed Automated Information Extraction (AIE) framework with a naive LLM-based approach across all three datasets (Figure 2(a) , Figure 3 ). AIE consistently outperforms the naive method, demonstrating its effectiveness in extracting information from diverse HLDs. Notably, AIE’s accuracy advantage widens under stricter RETA levels on FINE, highlighting its ability to achieve higher precision. To assess AIE’s adaptability across LLMs, we evaluated it using GPT-4, the current

Read the complete text