Hierarchical Robust PCA for Scalable Data Quality Monitoring in Multi-level Aggregation Pipelines
Preetam Kumar Ojha · 2025
Read Hierarchical Robust PCA for Scalable Data Quality Monitoring in Multi-level Aggregation Pipelines by Preetam Kumar Ojha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14524.txt.
Description
Preetam Kumar Ojha presents a method for spotting bad data in large pipelines where totals are built up level by level. The 2025 paper notes that fixed check rules often fail to scale across users, sessions, accounts, and regions. His approach applies a robust form of principal component analysis at each level of the stack.
At each level, the method splits the data into a clean low-rank part and a sparse part that holds the odd values. Those odd values flag errors, outliers, and totals that do not line up. Tests on made-up data sets with planted faults show the method beats rule-based checks at finding corruption. The design is modular, so it can grow with the pipeline.
Catalog record
| CREATOR | Preetam Kumar Ojha |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 11 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-14524