Automated Big Data Quality Assessment using Knowledge Graph Embeddings
Hadi Fadlallah, Rima Kilany, Mitri Haber, Ali Jaber · 2026
Read Automated Big Data Quality Assessment using Knowledge Graph Embeddings by Hadi Fadlallah, Rima Kilany, Mitri Haber, Ali Jaber (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18833.txt.
Introduction
Hadi Fadlallah and three colleagues assess big data quality with knowledge graph embeddings. Existing tools struggle to judge quality in context, leaning on strict matching that misses dataset traits. Their method predicts missing links between a dataset context and quality rules and dimensions in a graph of contextual traits and assessment steps. Weighted by edge attributes, the predictions form a tailored assessment plan. A radiation sensor case study shows the plan can be built for a real dataset.
Catalog record
| CREATOR | Hadi Fadlallah, Rima Kilany, Mitri Haber, Ali Jaber |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47361 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18833
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automated Big Data Quality Assessment using Knowledge Graph Embeddings Hadi Fadlallah, Rima Kilany, Mitri Haber, Ali Jaber Abstract Automated data quality assessment is crucial for managing big data, but existing solutions face challenges in achieving accurate context-aware assessment. This paper presents a novel knowledge-based approach to enhance automated data quality assessment. Our approach utilizes knowledge graph embeddings to predict missing edges between the input dataset’s context representation and the relevant quality rules and dimensions within a knowledge graph representing contextual data characteristics and the required quality assessment operations. We surpass conventional practices by integrating diverse representations within the knowledge graph, drawing insights from contextual information from a thorough literature investigation. This integration allows us to develop a comprehensive and context-specific data quality assessment plan tailored to each context. Leveraging the knowledge graph improves our understanding of the input dataset’s context, overcoming the limitations of traditional methods that rely solely on strict matching and overlook contextual characteristics. By injecting numerical edge attributes, we assign corresponding weights to each predicted quality measurement, providing a comprehensive data quality assessment plan for the input dataset. To evaluate our approach, we leverage AmpliGraph, a framework developed and benchmarked by AccentureLabs. The evaluation involves employing a real-world radiation sensors dataset provided by the Lebanese Atomic Energy Commission (LAEC-CNRS). The results obtained from this evaluation demonstrate the capability of our solution to generate a comprehensive data quality assessment plan for the given input dataset. K eywords Data quality assessment \cdot Data context \cdot Big data \cdot Machine learning \cdot Knowledge graph embeddings \cdot Automation 1 Introduction Data quality assessment is critical in ensuring data reliability, validity, and usability for various data-driven applications 38 . It serves as a fundamental step in data management, where different aspects such as data accuracy, completeness, consistency, and timeliness are assessed to make informed decisions and drive meaningful insights 44 . Manual data quality inspection becomes increasingly challenging when faced with big data due to the wide variety, high velocity, and sheer volume to be examined 20 . The human capacity to examine each data set and identify potential errors or inconsistencies, and develop the relevant quality assessment tasks becomes severely constrained. Moreover, this is highly time-consuming, often requiring significant resources and effort. Automated data quality assessment becomes paramount in the era of big data, where data volume, velocity, and variety are continuously growing. Automated approaches leverage computational algorithms, techniques, and tools to assess data quality attributes systematically and efficiently. Organizations can save time, reduce costs, and ensure a more consistent and objective data quality evaluation across large datasets by automating the process 14 . To address these challenges, we previously introduced BIGQA 13 , a comprehensive framework for automating big data quality assessment. BIGQA fills a gap in current big data quality assessment solutions and makes valuable contributions; the framework introduces a declarative approach that effectively automates data quality assessment operations while minimizing user intervention. By incorporating the ISO/IEC 15939 25 and ISO/IEC 25000 24 standards, it successfully addresses the unique challenges of big data’s volume, variety, and velocity. Additionally, BIGQA presents an innovative architectural design and implementation using prominent big data tools. The solution’s architecture aligns with the ISO/IEC 20547 26 big data reference architecture, enhancing its robustness and compatibility. Moreover, the framework offers a powerful capability of generating customized data quality reports, providing users with tailored insights. Most importantly, BIGQA has context-aware quality assessment capabilities since it incorporates a context analyzer component that harnesses the characteristics of the data context. It employs machine learning techniques, specifically Node2Vec 19 and K-Nearest Neighbors 37 to retrieve the most relevant data quality assessment plan from a knowledge graph that contains information about data contexts and their relevant quality assessment plans. Still, the context analyzer has limitations in considering the complete range of context characteristics. Moreover, the current implementation of the context analyzer in BIGQA focuses solely on selecting the quality assessment plan for the most similar data context in the knowledge graph. Focusing only on the most similar context could result in losing important details required to assess the incoming dataset accurately. The knowledge graph may also contain other data contexts providing additional information that could help create a more accurate and thorough assessment approach for the input dataset. To overcome these limitations, this paper focuses on enhancing the steps of gathering context information and generating a tailored data quality assessment plan to ensure accurate and context-specific execution (Figure 1 ); once a plan is generated, it will be converted to an executable code that will generate as expected the needed data quality report. This paper begins by investigating the literature to identify a comprehensive description of data context characteristics and develop the most complete and accurate representation of the data context. Data contexts and their attributes, including data size, structure, attribute types, domain, and file formats, are described in the knowledge graph. A dedicated data quality assessment plan is integrated and linked with the corresponding context characteristics by data experts or researchers. Each plan comprises several quality rules (base measures) and dimensions (derived measures). When dealing with a new data set, the context information is extracted and added to the knowledge graph; however, the link to a valid data quality assessment plan does not exist. This paper presents an enhanced approach that leverages knowledge graph embedding to predict missing edges between the input data context representation and the existing quality rules and dimensions within the knowledge graph. Our proposed method involves numerical edge injection attributes into knowledge graph embeddings to add weights to the predicted quality measurements, resulting in a complete data quality assessment plan. Injecting numeric edge attributes into knowledge graph embeddings is essential as it allows the model to capture additional information related to uncertainty, edge importance, and out-of-band knowledge. This enhancement improves the model’s predictive capabilities and enables it to generate more accurate and context-specific data quality assessment plans. Integrating literature-derived context characteristics into our methodology enables us to overcome potential disparities between the input dataset and the retrieved quality assessment plan’s schema, resulting in improved accuracy and effectiveness of the assessment process. Figure 1: Gathering context information and generating a data quality assessment plan We evaluated our solution using a Neo4j 1 1 1 https://neo4j.com knowledge graph representing the data context characteristics and the related data quality assessments alongside a knowledge graph embedding solution called AmpliGraph 8 developed by Accenture Labs. The results demonstrate that our approach provides a more accurate data quality assessment plan compared to the one retrieved by the BIGQA context analyzer component. The evaluation was conducted on a real-world dataset from radiation sensors provided by the Lebanese Atomic Energy Commission (LAEC-CNRS) 2 2 2 http://www.laec-cnrs.gov.lb/ . The remainder of this paper is organized as follows. Section 2 presents an overview of the fundamental concepts that form the basis of this research. Section 3 provides a literature review on knowledge graph embedding and existing automated data quality assessment approaches. Section 4 describes the proposed solution. Section 5 presents the experiment and evaluation, including a description of the datasets used and a comparison of the proposed method with BIGQA. Section 6 concludes the paper and presents future research directions. 2 Background This section overviews knowledge