Zero-Scan Data Quality: Leveraging Table Format Metadata for Continuous Observability at Scale Conference: 1st International Workshop on Data FORMATS for Modern Architectures and Workloads; May 31-June 05, 2026; Bengaluru, India 1st International Workshop on Data FORMATS for Modern Architectures and Workloads (FORMATS ’26), May 31-June 05, 2026, Bengaluru, India DOI: 10.1145/3802514.3812601 ISBN: 979-8-4007-2635-4/2026/05 CCS: Information systems Data management strategies CCS: Information systems Data quality

Mohit Verma, Shantanu Rawat, Christian Bush, Sumedh Sakdeo, Lokesh Amarnath Ravindranathan, Dwarak Bakshi · 2026

Read Zero-Scan Data Quality: Leveraging Table Format Metadata for Continuous Observability at Scale Conference: 1st International Workshop on Data FORMATS for Modern Architectures and Workloads; May 31-June 05, 2026; Bengaluru, India 1st International Workshop on Data FORMATS for Modern Architectures and Workloads (FORMATS ’26), May 31-June 05, 2026, Bengaluru, India DOI: 10.1145/3802514.3812601 ISBN: 979-8-4007-2635-4/2026/05 CCS: Information systems Data management strategies CCS: Information systems Data quality by Mohit Verma, Shantanu Rawat, Christian Bush, Sumedh Sakdeo, Lokesh Amarnath Ravindranathan, Dwarak Bakshi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30308.txt.

Introduction

Mohit Verma and five co-authors describe a metadata-first approach repurposing write-time table statistics for data quality observability without scanning data. Modern formats such as Apache Iceberg record timestamps, counts, value bounds, and null counts when writing. At LinkedIn across 200,000+ Iceberg tables, this satisfies about 60% of quality rules at zero compute cost and cuts profiling use by around 50%. Counters and sketches can raise coverage to close to 90% of rules, and they propose storing sketches in Puffin sidecar files.

Catalog record

CREATORMohit Verma, Shantanu Rawat, Christian Bush, Sumedh Sakdeo, Lokesh Amarnath Ravindranathan, Dwarak Bakshi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21791
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30308

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero-Scan Data Quality: Leveraging Table Format Metadata for Continuous Observability at Scale Conference: 1st International Workshop on Data FORMATS for Modern Architectures and Workloads; May 31-June 05, 2026; Bengaluru, India 1st International Workshop on Data FORMATS for Modern Architectures and Workloads (FORMATS ’26), May 31-June 05, 2026, Bengaluru, India DOI: 10.1145/3802514.3812601 ISBN: 979-8-4007-2635-4/2026/05 CCS: Information systems Data management strategies CCS: Information systems Data quality Mohit Verma, Shantanu Rawat, Christian Bush, Sumedh Sakdeo, Lokesh Amarnath Ravindranathan, Dwarak Bakshi Abstract . Modern table formats such as Apache Iceberg compute and store metadata—commit timestamps, record counts, and column-level statistics such as null counts and value bounds—at write time as part of file writing. These statistics serve query planning, yet they overlap substantially with data quality (DQ) monitoring needs. We describe a metadata-first approach that repurposes write-time statistics for continuous DQ observability—anomaly detection, drift monitoring, null-rate tracking—without scanning any data. Deployed at LinkedIn across 200,000+ Iceberg tables (800+ PB), this approach satisfies approximately 60% of user-defined DQ rules at zero marginal compute cost and reduces profiling resource consumption by around 50%. Extending manifest statistics with lightweight counters (sum, zero-value counts, boolean counts) and incrementally mergeable sketches—Theta sketches for distinct counts, KLL sketches for quantiles—can further raise metadata-satisfiable coverage to close to 90% of production DQ rules. We validate sketch accuracy, mergeability, and storage overhead on production data and propose that table formats should store per-file sketches in Puffin sidecar files, following the same store-then-aggregate pattern used for existing manifest statistics. † † cc-license: by 1. Introduction Data quality monitoring is essential for data-intensive organizations, yet the dominant paradigm treats it as a read-time concern: tools such as dbt tests ( dbt ) , Great Expectations ( ge ) , and Soda Core ( soda ) execute SQL queries or full table scans to validate expectations after data is written. At lakehouse scale—hundreds of thousands of tables, petabytes of data, continuous ingestion—ensuring every consumer has the right checks is impractical. To compensate, centralized profiling jobs run asynchronously, but this is prohibitively expensive to operate at scale. The cost shows up in two ways. First, scan-based validation : every existing DQ tool requires reading actual data rows, consuming cluster resources proportional to data volume. In our environment, scan-based profiling for just \sim 3,000 tables consumed \sim 1,600 TBhr of compute and read \sim 1,700 TB from HDFS daily. Second, detection latency : because scans run asynchronously, bad data propagates downstream before issues surface. Modern table formats already solve a significant portion of this problem. Apache Iceberg persists per-data-file column statistics in manifest files—record counts, null counts, NaN counts, bounds, and column sizes ( iceberg ) —as byproducts of Parquet/ORC encoding with negligible overhead. Delta Lake stores analogous statistics in its transaction log ( delta ) . Yet no existing tool exploits these statistics for quality monitoring. The metadata-first approach benefits organizations of any size: write-time statistics are free regardless of fleet size, and smaller organizations that lack dedicated profiling infrastructure benefit disproportionately. Our contributions are: (1) we quantify the overlap between table format metadata and DQ rules—approximately 60% from manifest statistics alone, rising to \sim 90% with counter and sketch extensions (§ 2.3 ); (2) we describe a production architecture that transforms Iceberg commit-time metadata into time-series observability signals; (3) we report deployment experience at 200,000+ table scale; and (4) we validate sketch-based extensions on production data and propose concrete format changes. 2. Motivation 2.1. Available Metadata in Table Formats Each data file registered in an Iceberg manifest —the metadata layer that tracks which files comprise a table snapshot—carries per-file statistics: record_count , null_value_counts , nan_value_counts , lower_bounds , upper_bounds , value_counts , and column_sizes ( iceberg ) . These are populated during Parquet or ORC file writing and serialized in Avro manifest files. Engines expose them via metadata tables—Spark ( table.files , table.snapshots ) and Trino ( $files , $partitions )—primarily for predicate pushdown, data skipping, and time travel. The same statistics that enable query optimization also enable data quality monitoring—a dual benefit that justifies compute investment at write time (§ 5.6 ). Delta Lake and Apache Hudi store analogous per-file statistics in their transaction log and metadata table, respectively ( delta ; hudi ) —the pattern is format-general. 2.2. The Gap: Existing DQ Tools Ignore Format Metadata Popular production-grade DQ tools operate by scanning data. dbt tests ( dbt ) use SQL SELECT queries returning failing rows; Great Expectations ( ge ) evaluates via SQL or Pandas; Databricks ( databricks ) processes complete tables per refresh; and Starburst Galaxy ( starburst ) relies on custom SQL rules rather than metadata. Even Hudi pre-commit validators ( hudi ) —the closest to write-time DQ—execute SQL queries against staged data before commit, remaining scan-based. Write-time statistics are computed universally, persisted durably, and queryable cheaply—yet the entire DQ ecosystem ignores them. 2.3. What Metadata Can and Cannot Do Manifest statistics are per-file, not pre-aggregated across partitions or tables—combining them requires reading manifest entries. The properties they capture are counts, nulls, and bounds, but not distributional properties (percentiles, distinct counts), referential integrity, or arbitrary SQL predicates. To quantify coverage, we analyzed \sim 15,000 user-defined DQ rules authored by data producers across \sim 1,800 datasets. Separately, data consumers evaluate \sim 22,000 rules daily across \sim 8,500 dataset evaluations within their own pipelines. Both populations show a consistent pattern (Table 1 ). Table 1. DQ rule coverage by metadata source. Rule Category Cons. Prod. Example Source Row-count bounds 15% 15% count>1000 Manifest Null checks 28% 26% notNull(c,<5%) Manifest Min/max, range 8% 11% min(age)>=0 Manifest Base manifest 51% 52% Compare metadata † 11% 14% compare(cnt,..) Manifest All manifest 62% 67% Sum, mean, zero/true 17% 8% sum(col)>0 Counter ext. Distinct count 8% 15% distinct(id) Theta sketch Median, pctl, IQR 1% < 1% median(c)<500 KLL sketch Total zero-scan \sim 88% \sim 90% Expr, regex, enum, arr. \sim 12% \sim 10% expr(a<b*1.15) Scan req. Cons. = consumer rules (downstream pipelines). Prod. = producer rules (data owners). † Compare manifest stats (counts, bounds, nulls) across snapshots/partitions. Our system already evaluates cross-snapshot comparisons (e.g., compare_count ) from time-series manifest statistics, bringing manifest coverage to \sim 62–67%. The “counter extensions” row covers metrics such as sum, zero-value count, and boolean counts—not yet in Iceberg manifests but representable as O(1) per-row counters with the same write-time overhead as existing null counts (§ 5.1 ). Distinct-count checks—the largest gain for producer rules—are addressable via Theta sketches (§ 5.2 ). Additionally, \sim 3,000 user-defined checks monitor data freshness—also surfaceable entirely from Iceberg commit-time metadata. Analyzing historically triggered DQ failures reinforces this: metadata-satisfiable rules account for over 50% of all consumer-side assertion failures observed in production, driven primarily by row-count and null assertions. Limitations beyond rule coverage. Four structural gaps remain: (1) merge-on-read tables (§ 5.3 ); (2) wide tables with thousands of columns (§ 5.5 ); (3) metrics such as distinct counts and quantiles require sketches (§ 5.2 ); (4) unmaterialized views have no underlying data files—an inherent boundary. 3. System Architecture Figure 1. Iceberg commit metadata flows through Kafka to consumers that compute arrival and column-level statistics, feeding anomaly and drift detection. Writer identity. System operations such as compaction, sorting, and purging create new snapshots without changing underlying data. Without

Read the complete text