Advanced ingestion process powered by LLM parsing for RAG system
Author: Arnau Perez Perez, Advisor: Xavier Vizcaino Gascon · 2024
Read Advanced ingestion process powered by LLM parsing for RAG system by Author: Arnau Perez Perez, Advisor: Xavier Vizcaino Gascon (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15262.txt.
Introduction
Arnau Perez presents a thesis on feeding complex documents into retrieval-augmented generation systems, supervised by Xavier Vizcaino, from 2024. Such systems stumble on mixed files: slides, dense reports, scans. The design uses language-model-powered optical character recognition with several parsing strategies, pulling content into linked nodes that keep relationships between text, tables and images.
A multimodal assembler agent and a flexible embedding plan turn the parsed nodes into searchable memory with context-aware metadata. Tests across several knowledge bases showed better answer relevance and faithfulness than plainer pipelines. The work targets the unglamorous ingestion step where retrieval quality is largely decided.
Catalog record
| CREATOR | Author: Arnau Perez Perez, Advisor: Xavier Vizcaino Gascon |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15262
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Advanced ingestion process powered by LLM parsing for RAG system Author: Arnau Perez Perez, Advisor: Xavier Vizcaino Gascon Abstract Abstract: Retrieval Augmented Generation (RAG) systems struggle with processing multimodal documents of varying structural complexity. This paper introduces a novel multi-strategy parsing approach using LLM-powered OCR to extract content from diverse document types, including presentations and high text density files both scanned or not. The methodology employs a node-based extraction technique that creates relationships between different information types and generates context-aware metadata. By implementing a Multimodal Assembler Agent and a flexible embedding strategy, the system enhances document comprehension and retrieval capabilities. Experimental evaluations across multiple knowledge bases demonstrate the approach’s effectiveness, showing improvements in answer relevancy and information faithfulness. I Introduction The field of Natural Language Processing has witnessed remarkable progress with the advent of Large Language Models (LLMs). These models have demonstrated unprecedented capabilities in processing multimodal data, including text and images. However, a significant challenge in LLM implementation remains the context window constraint, which limits the volume of information processed concurrently. Leading providers have made substantial strides in expanding context limitations, with some models now capable of handling up to 2.000,000 tokens simultaneously Gemini2M . Nevertheless, this expansion comes at a cost: increased context length correlates with longer time to first token (TTFT), potentially impacting model performance and user experience Context . Recent work by Anthropic has explored caching techniques to mitigate this issue and reduce TTFT AnthropicCaching . To further enhance LLM efficiency and overcome these challenges, researchers introduced the Retrieval Augmented Generation (RAG) system. In this case for file data. A critical component of RAG is the ingestion phase, where the method of chunking plays a pivotal role in determining the quality of subsequently retrieved data. Common strategies include semantic, recursive, and hierarchical splitting. However, these approaches often fail to account for the diverse data structures present in source materials, such as images, tables, headers and pages. Moreover, they can introduce computational challenges Semantic . This paper presents a complex ingestion process that not only considers the various data structures within files but also incorporates relevant metadata and establishes hierarchical relationships to improve retrieval. II Ingestion \begin{overpic}[preprocessing.png] \end{overpic} Figure 1: Preprocessing pipeline for document ingestion in RAG system. The flowchart illustrates the parsing and assembling process for PDF, DOCX, and PPTX files. The pipeline incorporates FAST, OCR, and LLM parsing techniques, followed by image description, text extraction, and snapshot creation. The assembler combines these elements to produce a markdown file per page and a concatenation of them. II.1 Pre-processing The pre-processing stage consists of three subprocesses: parsing, assembling and metadata extraction (see Figure 1 ). The parsing phase extracts image and text content using three strategies: 1. FAST : Utilizes Python libraries to extract text and images from each document page. 2. LLM : Employs a multimodal LLM for OCR task, extracting text, table content, and image information. It describes images without text or extracts text from images. In this case it was used Sonnet 3.5 v2 model from Anthropic Claude family AnthropicModels . 3. OCR : Leverages external and dedicated machine learning models for OCR task. In this case it was used the AWS Textract service Textract . Before assembly, images are analyzed and described based on content type. For plots, the system extracts axis values, legends, labels, and provides a visualization description. Flowcharts are described in terms of process relationships. Page snapshots are then captured. In the assembling process, a Multimodal Assembler Agent integrates the page snapshot, images with descriptions, and text extracted from all three strategies for each page. The agent produces a synthesized markdown file. The output includes individual page markdowns, snapshots, and described images, which are concatenated into a comprehensive document-level markdown. The metadata extraction phase employs a Metadata Extractor Agent to analyze the consolidated markdown, extracting fields such as topic, keywords and summary. The system also extracts metadata directly from the document (title, author, creation date, last modified date, etc.), providing a comprehensive information set about the processed document. II.2 Processing \begin{overpic}[processing.png] \end{overpic} Figure 2: The illustration shows the assembling of the page 2 of the paper AIDAPaper . There are 5 kind of nodes represented: Header, Text, Table, Image and Page. Note that the image node is represented in the markdown file where the src is the id of image extracted plus its extension and the alt is the description of the image. Following the segmentation of document content into markdown files, including snapshots, images, and metadata, the processing phase extracts information nodes from this data. This approach aims to represent information in a vectorial space, with the primary challenge being the extraction of relevant information and determining its optimal embedding representation. Figure 2 illustrates four distinct node types extractable from a single page ( Page node): Header , Text , Table and Image nodes. Text Nodes can encompass various forms of textual information, including paragraphs, words, sentences, ordered lists, and bullet point lists, each representing distinct information requiring specific consideration. The processing phase comprises two primary subprocesses: Splitting and Contextualization . During Splitting, markdown files are parsed and segmented into the aforementioned node types. Four relationship types are established between nodes: next, previous, parent and child. Only Header nodes can possess child nodes, facilitating context window tracking and hierarchical structuring. For Text nodes exceeding a size threshold, a Recursive Splitter or Semantic Splitter is employed for chunking. Contextualization focuses on Table and Header nodes. A Question Generator Agent creates a set of questions answerable using the information within the table or header node content. Subsequently, a Summary Generator Agent produces context-aware summaries for both table and header nodes. Following lower-level information node creation, the system generates Page and Document nodes, connecting previously established nodes to these higher-level nodes. The Page node contains the individual page’s markdown representation, while the Document node aggregates all page markdowns. Page nodes are also linked to their respective Document node. In the final stage, the Summary Generator Agent is reemployed to generate summaries for both the Document node (if not previously extracted during preprocessing) and individual Page nodes. This hierarchical approach to information extraction and representation enables comprehensive and context-aware structuring of document content, facilitating more effective retrieval and utilization in downstream RAG system processes. II.3 Embedding At this juncture, all components are prepared for integration. Three critical decisions must be made: selecting a vector database, choosing an embedding model and determining the data to be embedded. The choice of vector database depends on various factors, with numerous options available in the industry. Pinecone, specifically designed for embedding processes in RAG systems, offers optimized performance for this task. Its namespace system provides excellent scalability, and it is engineered for rapid query execution. However, it has a relatively low metadata storage capacity (40KB) PineconeMetadata , which may be a limiting factor depending on the volume of metadata generated. Alternatively, OpenSearch, an open-source solution reimagined as a vector database, offers a hybrid approach. It combines vector and ordinal database functionalities, enabling conventional queries unrelated to vector data. In contrast to Pinecone, OpenSearch does not impose a strict distinction between metadata and the vector itself, thus removing specific storage limitations. The selection of an embedding model is