Accelerating Fresh Data Exploration with Fluid ETL Pipelines

Maxwell Norfolk, Dong Xie · 2026

Read Accelerating Fresh Data Exploration with Fluid ETL Pipelines by Maxwell Norfolk, Dong Xie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22220.txt.

Description

Norfolk and Xie propose fluid ETL pipelines for exploring data while it is still arriving. Analysts often need to scan fresh data for main traits or odd values. Classic pipelines prep data with many routines, like parsing and indexing, to cover unknown goals. That prep work is costly. Fluid pipelines can start and stop prep routines on demand, without blocking ingestion. They run extra routines when idle or cheap spot capacity is free. Tests on a real dataset show the idea works.

Catalog record

CREATORMaxwell Norfolk, Dong Xie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35584
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22220

Browse the full library · About the Tower of Babel