Hierarchical Robust PCA for Scalable Data Quality Monitoring in Multi-level Aggregation Pipelines

Preetam Kumar Ojha · 2025

Read Hierarchical Robust PCA for Scalable Data Quality Monitoring in Multi-level Aggregation Pipelines by Preetam Kumar Ojha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14524.txt.

Description

Preetam Kumar Ojha presents a method for spotting bad data in large pipelines where totals are built up level by level. The 2025 paper notes that fixed check rules often fail to scale across users, sessions, accounts, and regions. His approach applies a robust form of principal component analysis at each level of the stack.

At each level, the method splits the data into a clean low-rank part and a sparse part that holds the odd values. Those odd values flag errors, outliers, and totals that do not line up. Tests on made-up data sets with planted faults show the method beats rule-based checks at finding corruption. The design is modular, so it can grow with the pipeline.

Catalog record

CREATORPreetam Kumar Ojha
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE11 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14524

Browse the full library · About the Tower of Babel