Declarative Data Services: Structured Agentic Discovery for Composing Data Systems

Shanshan Ye, Duo Lu · 2026

Read Declarative Data Services: Structured Agentic Discovery for Composing Data Systems by Shanshan Ye, Duo Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20690.txt.

Introduction

Shanshan Ye and Duo Lu propose Declarative Data Services, a structured way for agents to compose data systems from user intent. Unbounded agents that iterate on failure logs often fail to converge on a working stack, even with added knowledge. Their framework splits the search into bounded parts with four typed contracts, covering intent, operator graphs, per-system skills, and runtime attribution. Sub-agents scan each typed space while the framework holds the whole together.

Catalog record

CREATORShanshan Ye, Duo Lu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74707
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20690

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Declarative Data Services: Structured Agentic Discovery for Composing Data Systems Shanshan Ye, Duo Lu Abstract Agentic discovery has shown that LLM-driven search can find novel algorithms, designs, and code under benchmark conditions. Translating the paradigm to multi-system data backends surfaces a harder problem: the search space is heterogeneous, the verifier is whether a deployed stack actually runs, and composition knowledge is unevenly captured in pretraining. Unbounded agentic discovery , a coding agent iterating on failure-log feedback, fails to converge consistently on a working stack even when iteration and explicit composition knowledge are added. We propose Declarative Data Services (DDS), an architecture for structured agentic discovery of data-system compositions from declarative user intent. The framework owns four typed contracts at successive layers (intent, operator DAG, per-system skills, runtime attribution) that decompose the global search into bounded sub-searches; sub-agents search each typed space, while the framework provides the channels by which knowledge flows forward as inline skill citations and errors route backward as typed signals. As a proof of life on a trading-backend workload, DDS converges where unbounded discovery does not; runtime failures become skill patches that the next deployment cites inline. We position this as an early prototype reporting lessons from real-world data-system composition. 1 Introduction Agentic discovery has made a fast transition from research to product. AlphaEvolve [ 39 ] discovered novel algorithms by LLM-driven evolutionary search; EvoX [ 36 ] and AdaEvolve [ 8 ] meta-evolved the search strategies themselves for systems-discovery problems; GEPA [ 1 ] optimized declarative LM-module graphs by reflective natural-language attribution; Glia [ 25 ] produced expert-level distributed-systems designs by multi-agent reasoning; agentic coding tools such as Claude Code [ 3 ] moved single-repository code generation into product-grade workflows. The common shape across these systems is structured agentic search : an LLM-driven agent explores a typed solution space, a verifier evaluates each candidate, and the loop refines under structured feedback. Translating this paradigm to multi-system data backends, the kind of stack a small team or individual without a dedicated data-engineering function (i.e., a solo trader, a small product team, a research group) would compose to run a real workload, surfaces a different set of problems. The scope is end-to-end: composing across queues, OLAP and OLTP stores, caches, search indices, and the connectors and configurations between them, then deploying that composition as a working stack and evolving it as the workload moves. The search space is heterogeneous; the verifier is whether the deployed stack actually runs and meets declared SLOs, not a benchmark answer key; ground truth is partial; and composition knowledge that distinguishes a working stack from a broken one is unevenly captured in pretraining and changes with every release cycle. Adjacent automation searches narrower spaces: infrastructure-as-code renders a chosen architecture, self-driving DBs tune inside one product, and modern data-stack tools declare within one pipeline stage. The search above these (which topology, which products, which configurations actually work together) is the gap DDS targets. The question is not whether agents can codegen, but how the structure around them should be designed: how should typed abstractions and coding agents split responsibility to discover, deploy, and evolve data backends from user intent? Figure 1: End-to-end view of DDS. The user states intent in natural language with concrete constraints; DDS sits between intent and deployment as a structured-discovery framework; the output is a multi-system backend whose components specialize across the workload. Figure 1 sketches our answer. The framework owns typed contracts at four layers (L1–L4); sub-agents search each layer’s typed sub-space. Each contract is validatable , citable , and editable , properties that let knowledge pass forward through the layers as the search progresses and let runtime errors route backward to the layer owning the violated decision; § 3 develops the architecture. We make three contributions. (i) An architecture for structured agentic discovery of multi-system data backends: a framework/agent split with four typed contracts at L1–L4, governed by two ownership rules (the framework owns each layer’s contract and validation; sub-agents own the bounded search inside that contract; § 3 ). (ii) Two architectural ideas that make discovery structured rather than unbounded: typed attribution at L4 routes every runtime signal to the layer owning the violated decision, and agent skills are the persistent memory where composition knowledge accumulates (§ 3 , § 4.3 ). (iii) A trading-backend case study (§ 4 , proof of life), with skill-content ablation and learning-loop demonstration; a second domain (chat, Appendix D ) describes operator-algebra extensibility. 2 Motivation and Scope A running scenario. We anchor the discussion to the trader workload of Fig. 1 (full intent in § 4 ): a real-time analytics backend combining high-throughput streaming ingest, multi-year time-series history, low-latency operational lookups, and a small budget. The user turns to a coding agent such as Claude Code; on a single repository the agent succeeds, on this multi-system stack it fails to converge consistently (empirical evidence in § 4.2 and Appendix E ). This is one instance of a broader class: agentic discovery for real-world data-system composition, where the search space is heterogeneous and the verifier is whether a deployed stack actually runs. Adjacent work covers slices, not structured discovery. Adjacent directions search narrower spaces than DDS targets. Polystores [ 20 ] search at query time across an already-composed set of stores, not the composition itself. Self-driving DBs [ 52 ] search the configuration space within one product, not across products. Modern data-stack tools (dbt [ 18 ] , Airbyte [ 2 ] , Fivetran [ 24 ] ) offer declarative surfaces within one pipeline stage but do not compose cross-stage dataflow. Vendor data platforms (Snowflake [ 16 ] , Databricks Lakehouse [ 6 ] ) lock the user into one vendor’s product mix and are not neutral over the 400+ database systems in production [ 17 ] . Agent-first data-system redesign [ 38 ] redesigns databases for agents; the complementary direction, agent-driven discovery of multi-system backends from intent, is what DDS targets. Infrastructure-as-code with LLMs (Pulumi-AI [ 45 ] , Terraform-AI [ 51 ] , IaC-Eval [ 30 ] , MACOG [ 27 ] ) renders a chosen architecture within one IaC dialect; we sit above that, with any such IaC backend a physical-layer implementation under our L3–L4 contracts (extended treatment in Appendix J ). Composition demands typed contracts, and composition knowledge demands persistent memory. User intent carries constraints an agent cannot reliably infer unaided, along six typed dimensions (data model, access pattern, scale, latency, consistency, and cost; elaborated in § 3 ) that mirror the textbook view of a data-intensive application [ 29 ] . “One size fits all” is settled [ 50 ] , so every realistic intent forces composition. What agents need is not more context but typed contracts that bound the search: a validated intent, a type-checked operator DAG, skill contracts per system, and layer-attributed runtime signals. Empirical work on agent reliability supports this [ 9 , 41 ] : typed failures should route to the layer owning the violated decision rather than rely on free-form coordination. Composition knowledge is not just absent but unstable; connector configurations, recommended images, and version-specific quirks change with every release. A framework that hard-codes composition rules ages out within a release; a

Read the complete text