Method for aggregating unstructured data using LLM
Lazebnyi Vsevolod, Tereshkina Natalia, Maria Shabarina, Dmitriy Fedorov · 2026
Read Method for aggregating unstructured data using LLM by Lazebnyi Vsevolod, Tereshkina Natalia, Maria Shabarina, Dmitriy Fedorov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16425.txt.
Introduction
Lazebnyi and colleagues present a pipeline for collecting and merging web data that keeps working when pages change. Hybrid scraping handles static pages and pages that load late, results land in MongoDB, and language models pull entries into a fixed JSON form. A two-stage check compares embeddings from several model outputs at different temperatures, plus consistency rules, to catch invented facts. Experiments reported high accuracy on key fields and held up when page layouts changed, suiting news aggregation and near-real-time monitoring.
Catalog record
| CREATOR | Lazebnyi Vsevolod, Tereshkina Natalia, Maria Shabarina, Dmitriy Fedorov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28168 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.16425
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Method for aggregating unstructured data using LLM Lazebnyi Vsevolod, Tereshkina Natalia, Maria Shabarina, Dmitriy Fedorov Abstract This paper presents a method for the automated collection and aggregation of unstructured data from diverse web sources, utilizing Large Language Models (LLMs). The primary challenge with existing techniques is their instability when the structure of webpages changes, their limited support for dynamically loaded content during information collection, and the requirement for labor-intensive manual design of data pre-processing processes. The proposed algorithm integrates hybrid web scraping (Goose3 for static pages and Selenium+WebDriver for dynamic ones), data storage in a non-relational MongoDB database management system (DBMS), and intelligent extraction and normalization of information using LLMs into a predetermined JSON schema. A key scientific contribution of this study is a two-stage verification process for the generated data, designed to eliminate potential hallucinations byy comparing the embeddings of multiple LLM outputs obtained with different temperature parameter values, combined with formalized rules for monitoring data consistency and integrity. The experimental findings indicate a high level of accuracy in the completion of key fields, as well as the robustness of the proposed methodology to changes in web page structures. This makes it suitable for use in tasks such as news content aggregation, monitoring, and log analysis in near real-time mode, with the capacity to scale rapidly in terms of the number of sources. Keywords LLM \cdot Data Aggregation \cdot Unstructured Data Processing \cdot Embedding \cdot Auto Validation \cdot JSON 1 Introduction To solve the problem of aggregating unstructured data from streaming sources, we propose a technique that automates the process of extracting and organizing web data using large language models (LLMs) [ 1 ] . Extracting information from diverse and dynamically changing web sources, including news portal sites, online marketplaces, and system logs, is becoming an essential task for business intelligence, market monitoring, and competitive analysis. Traditional data extraction methods based on XPath/CSS selectors or specialized parsing tools are used for various purposes, such as monitoring prices from competitors, reviewing links on a webpage, transferring product details, and so on. However, they continue to be used to collect data from sources that do not have a permanentlyly predictable structure, such as media news articles and event announcements. In its raw form, this data does not have a common structure and requires more detailed processing. Standard collection and aggregation algorithms are often inadequate in such scenarios, as they are sensitive to changes in source data, necessitating additional time and effort to support parsing of each source and perform post-processing operations. The development of large language models (LLM) has provided novel opportunities for automating data extraction [ 2 ] . These models can interpret the context of a given text and generate structured responses, making them suitable for various tasks [ 3 ] . The integration of LLMs with retrieval-augmented generation (RAG) techniques and hybrid pipelines, such as ScrapeGraphAI and LangChain, enhances the flexibility, adaptability, and scalability of these systems [ 4 ] . This integration also reduces the need for manual configuration when dealing with static and dynamic web pages. However, applying LLMs directly to each page may result in high computational costs, and it does not solve the issues of data duplication and validation. Recent research in this area has led to the development of hybrid algorithms that integrate classical parsing techniques, deduplication using semantic embeddings [ 5 ] , intelligent data structuring with large language models (LLM), and two-stage verification of the generated data to prevent hallucinations – incorrect data output formats [ 6 ] . This approach enhances the system’s resilience to changes in the source data, ensuring high-quality and scalable solutions for processing unstructured web content in real-time. The effectiveness of this approach has been validated through experimental results in various subject areas. The shift towards LLM-based and hybrid architectures represents a promising direction for building sustainable and scalable systems capable of automatically extracting and organizing web data. 2 Overview of existing approaches 2.1 Comparison of approaches The various approaches and their methods are similar in intent, but at the same time, they have their own advantages and disadvantages. Table 1 below outlines a selection of methods that can be employed when carrying out similar tasks of collating unstructured data during streaming processing. Table 1: Comparison of specified tools and approaches. Approaches Method Advantages Disadvantages Parsing XPath/CSS-selectors High speed, simplicity, low cost Fragility to structure changes, does not work with dynamic content Regular expressions Flexibility, precision on known patterns Requires manual configuration, does not transfer well Headless browsers (Selenium, Playwright) Work with dynamic pages High cost, low speed, complexity of support Template methods Extraction automation, suitable for similar pages Limited versatility, sensitivity to structure Manual template generation High accuracy on small volumes Labor-intensive, does not scale Information extraction (IE) CRF, BERT, transformers High accuracy, resistance to local changes Requires markup, low portability Ontological methods Deep understanding, extensibility Requires expert, complex support Lexical patterns Easy implementation, transparency Limited flexibility Unsupervised methods No markup needed, automation Less accurate, harder interpretation Structuring using LLM GPT-4, Llama Versatility, deep understanding of context Cost, latency, hallucinations Integration with ontologies Improved accuracy Complex integration Combined approach Integration of parsing, IE, LLM Efficiency, accuracy, automation Complex implementation Data Mining ML methods Pattern detection, scalability Requires large datasets 2.2 Parsing Traditional web scraping tools, such as Beautiful Soup and Scrapy, are widely used for extracting data from static HTML pages due to their speed, low cost, and ease of implementation [ 7 ] . These tools rely on DOM tree analysis, using XPath/CSS selectors, as well as regular expressions, to ensure deterministic results with an unchanged page structure. However, these methods are not practical for websites with dynamically loaded content, such as single-page applications (SPAs) and sites with endless scrolling and JavaScript interactivity. For these types of websites, additional tools like Selenium and Playwright must be used [ 8 ] . Additionally, any changes to the layout or structure of target blocks can destabilize parsers, requiring manual intervention. This reduces the effectiveness of traditional scraping tools for long-term monitoring of diverse resources. 2.3 Information Extraction Advanced approaches utilize statistical and neural network models for the extraction of entities and relationships [ 9 ] . These techniques include BiLSTM-CRF, CRF taggers, and modern transformers that are fine-tuned on specialized datasets [ 10 ] . The main advantage of these approaches is their high accuracy in extracting information from familiar data, as well as their resistance to local text changes. However, such models require marked-up corpora for each new entity type or source. Additionally, the transfer of these models to new categories of data often results in a decrease in quality, limiting their versatility when dealing with highly variable source structures. 2.4 Structuring using LLM With the advent of large language models such as GPT-4, Claude and Llama, it has become possible to extract structured information from unstructured text using prompts, including a description of the target structure, examples and additional transformation rules [ 11 ] . This approach offers several advantages, including universality, zero- or few-shot learning capabilities, and a deep understanding of contextual meaning in text [ 12 ] . However, LLM-based methods have some drawbacks, including high resource and latency costs, stochastic outputs