Vidya: An AI-Driven Modular Pipeline for Archival Automation and Semantic Metadata Enrichment
Cloter Migliorini Filho 1, Julia Graciela Machado 1 · 2026
Read Vidya: An AI-Driven Modular Pipeline for Archival Automation and Semantic Metadata Enrichment by Cloter Migliorini Filho 1, Julia Graciela Machado 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16338.txt.
Introduction
Migliorini Filho and three colleagues built Vidya at LAMUHDI to meet NOBRADE and ISAD(G) while cutting archival processing from decades to days. The problem is dark data: digitized objects with too little metadata to be found or reused, while manual description stays slow and costly. Vidya orchestrates language models with free and open-source tools, binding each generation to YAML-defined ontologies and Pydantic validation, so probabilistic models yield structured JSON. The modular pipeline aims at low-cost use in memory institutions on modest hardware, following Maker principles.
Catalog record
| CREATOR | Cloter Migliorini Filho 1, Julia Graciela Machado 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22171 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16338
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vidya: An AI-Driven Modular Pipeline for Archival Automation and Semantic Metadata Enrichment Cloter Migliorini Filho 1, Julia Graciela Machado 1 Abstract Resumo The large-scale digitization of historical archives has created a paradox: “dark data”—digital objects lacking metadata for retrieval. Manual archival description is slow and expensive, limiting discovery and reuse. We propose Vidya, a modular pipeline that orchestrates Large Language Models (LLMs) and FOSS tools to automate semantic enrichment and archival ingestion at scale. Vidya constrains generations using YAML-defined ontologies and Pydantic validation, producing deterministic, structured JSON outputs from probabilistic models. Developed at Laboratory for Digital Humanities and Innovation (LAMUHDI) of the State University of Ponta Grossa (UEPG), Vidya applies Maker principles and open-source practices to enable low-cost deployment in memory institutions using modest hardware. We compare LLM performance and present a cost–benefit analysis showing major gains, reducing processing time from decades to days while complying with NOBRADE and ISAD(G). Resumo. A digitalização em larga escala de arquivos históricos criou um paradoxo: “dados escuros”—objetos digitais sem metadados para recuperação. A descrição arquivística manual é lenta e cara, limitando a descoberta e o reuso. Propomos o Vidya, um pipeline modular que orquestra Modelos de Linguagem de Grande Porte (LLMs) e ferramentas FOSS para automatizar o enriquecimento semântico e a ingestão arquivística em escala. O Vidya restringe as saídas por ontologias definidas em YAML e validação com Pydantic, produzindo saídas JSON determinísticas e estruturadas a partir de modelos probabilísticos. Desenvolvido no Laboratório de Humanidades Digitais e Inovação (LAMUHDI) da Universidade Estadual de Ponta Grossa (UEPG), o Vidya aplica princípios Maker e práticas de código aberto para viabilizar implantação de baixo custo em instituições de memória com hardware modesto. Comparamos o desempenho de diferentes LLMs e apresentamos uma análise de custo–benefício com ganhos expressivos, reduzindo o tempo de processamento de décadas para dias, com conformidade à NOBRADE e à ISAD(G). 1 Introduction The modern archival landscape faces a growing challenge: the massive digitization of historical archives has generated vast amounts of unstructured “dark data.” While essential for preservation, digitization often creates digital objects lacking the metadata required for proper retrieval and interpretation [ 11 ] . This valuable data remains largely inaccessible because manual archival description is prohibitively slow and expensive, exacerbating backlogs of unprocessed files [ 23 ] . Consequently, there is an urgent need for scalable and flexible systems capable of automating archival processes and enriching digitized materials. Traditional manual transcription methods are error-prone and labor-intensive, especially with large text volumes, frequently resulting in incomplete or inaccurate metadata [ 20 , 4 ] . As digitized collections expand, innovative approaches are essential to mitigate these descriptive challenges. Artificial Intelligence (AI), particularly Large Language Models (LLMs), offers significant promise for automating document ingestion and metadata generation. However, applying AI to archives presents unique challenges, including model hallucinations, inconsistent outputs, and difficulties ensuring standardized formats [ 7 , 14 ] . These issues necessitate a controlled, structured approach to leverage AI’s potential while maintaining the rigorous integrity and accuracy required for archival work. In response, this paper presents Vidya, a modular pipeline architecture designed to automate archival workflows by orchestrating LLMs for semantic enrichment. Vidya employs a “digital straitjacket” approach, using YAML-defined ontologies and Pydantic validation to enforce deterministic, structured JSON outputs, enabling interoperability with repository platforms [ 24 , 3 ] . This ensures that probabilistic models produce outputs strictly aligned with international archival standards, such as NOBRADE [ 1 ] and ISAD(G) [ 17 ] . Developed at the Laboratory for Digital Humanities and Innovation (LAMUHDI) at the Museu Campos Gerais (UEPG), Vidya combines Maker principles with the public university mission to democratize AI access in memory institutions using low-cost hardware and open-source software. This interdisciplinary approach fosters digital sovereignty and serves as a training platform where historians and computer scientists collaborate, narrowing the gap between archival theory and digital innovation. 2 Related Works The challenge of efficiently managing, enriching, and preserving digital collections has been the focus of several efforts in the archival field. Traditional methods of archival description have long been criticized for their inefficiency and the vast human labor required, which can lead to errors and inconsistencies [ 5 ] . As digital archives grow, there is an increasing need for automated solutions that streamline these processes without compromising data integrity. Recently, integrating Artificial Intelligence (AI) and Machine Learning (ML) into archival workflows has garnered attention as a potential solution [ 12 ] . The use of Large Language Models (LLMs) for automating archival processes has gained traction in recent literature. Karpov et al. (2024) proposed a system using deep learning models to generate metadata from historical texts, significantly improving retrieval times and accuracy for digital archives, while effectively addressing inconsistencies and missing metadata in digitized collections [ 9 ] . Furthermore, Johnson and McKenna (2023) emphasized the importance of supervised learning and neural network-based techniques to enhance the quality of metadata tags generated from historical documents, focusing on improving indexing and search accuracy in repositories [ 8 ] . Despite these benefits, applying AI directly to archives presents several challenges, especially concerning model hallucinations and standards compliance. Huang et al. (2025) reviewed the limitations of applying AI to archival tasks, noting the tendency of unconstrained language models to produce unreliable outputs. They proposed hybrid systems that integrate traditional metadata standards with AI to improve output reliability [ 6 ] . Similarly, Lee and Zhang (2023) stressed the critical importance of a structured validation framework to ensure AI outputs strictly conform to archival standards, such as ISAD(G) and NOBRADE [ 10 ] . The use of LLMs in historical research introduces significant ethical challenges regarding “algorithmic hallucinations.” Vidya addresses this through a “digital straitjacket,” implemented via description models structured around internationally recognized ontologies. By forcing the AI to adhere to strictly defined YAML schemas , Vidya ensures it operates as an assistant governed by human-defined rules. This approach safeguards the integrity of automated descriptions, ensuring they remain a consistent representation validated by humans. Integrating metadata standards with open-source repository systems is crucial for scalable archival solutions. Tainacan is widely adopted in Brazil for managing digital collections while supporting standards like METS [ 16 ] and PREMIS [ 19 ] , offering potential for integration with AI-based metadata tools [ 21 ] . Archivematica is recognized as a comprehensive preservation system for institutions complying with international standards, particularly integrating with OAIS to ensure long-term data integrity [ 13 ] . Similarly, DSpace provides a framework for managing and disseminating digital archives, increasingly focusing on integrating machine learning algorithms to automate document classification [ 2 ] . Vidya presents a significant advancement for data ingestion and semantic enrichment into systems like Archivematica, DSpace, Tainacan, and Omeka. While Archivematica offers comprehensive preservation workflows, it lacks the flexibility to integrate modern AI technologies. DSpace and Tainacan excel in repository management but require significant manual metadata entry and lack AI-driven automation. Omeka S supports metadata standards and customization, yet still requires manual involvement in metadata generation, hindering scalability for large collections. In contrast, Vidya integrates LLMs for automatic metadata generation, drastically reducing manual intervention while ensuring compliance with standards like ISAD(G) and NOBRADE. Its modular architecture allows institutions to incorporate AI without modifying