Governing Cloud Data Pipelines with Agentic AI
Aswathnarayan Muthukrishnan Kirubakaran, Adithya Parthasarathy, Nitin Saksena, Ram Sekhar Bodala, Akshay Deshpande, Suhas Malempati, Shiva Carimireddy, Abhirup Mazumder, [1ex] IEEE Senior Member, USA · 2025
Read Governing Cloud Data Pipelines with Agentic AI by Aswathnarayan Muthukrishnan Kirubakaran, Adithya Parthasarathy, Nitin Saksena, Ram Sekhar Bodala, Akshay Deshpande, Suhas Malempati, Shiva Carimireddy, Abhirup Mazumder, [1ex] IEEE Senior Member, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23737.txt.
Description
Aswathnarayan Kirubakaran and seven co-authors propose governing cloud data pipelines with bounded AI agents. Production pipelines face shifting workloads, evolving schemas, cost pressure, and strict compliance rules, yet most still run on fixed settings and reactive, manual operations, which slows recovery and wastes resources. Their architecture, Agentic Cloud Data Engineering, places specialized agents in the pipeline's control plane.
The agents read telemetry and metadata, reason against written cost and compliance policies, and propose limited actions such as reconfiguring resources, reconciling schemas, or recovering from failures. Every action is checked against policy before it runs, keeping behavior predictable and auditable. On representative batch and streaming workloads, the platform cut mean recovery time by up to forty-five percent, lowered operating cost by about a quarter. It also cut manual work by more than seventy percent.
Catalog record
| CREATOR | Aswathnarayan Muthukrishnan Kirubakaran, Adithya Parthasarathy, Nitin Saksena, Ram Sekhar Bodala, Akshay Deshpande, Suhas Malempati, Shiva Carimireddy, Abhirup Mazumder, [1ex] IEEE Senior Member, USA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-23737