Accelerating the Operation of Complex Workflows through Standard Data Interfaces
1 st Taylor Paul, 2 st William Regli · 2024
Read Accelerating the Operation of Complex Workflows through Standard Data Interfaces by 1 st Taylor Paul, 2 st William Regli (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13339.txt.
Introduction
Taylor Paul and William Regli argue for standard data channels in science workflows in a 2024 paper. Complex workflows are usually drawn as steps wired point to point, which limits reuse and portability. Their position paper proposes wiring steps through shared channels in a network message service instead. Data sharing would then separate cleanly from data processing.
They offer a first reference model, an architecture, and open tools that implement the idea today. The goal is a shared foundation layer for workflows across platforms and networks. They close by asking the workflow community to judge the approach and name the requirements that matter most.
Catalog record
| CREATOR | 1 st Taylor Paul, 2 st William Regli |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 8 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13339
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Accelerating the Operation of Complex Workflows through Standard Data Interfaces 1 st Taylor Paul, 2 st William Regli Abstract In this position paper we argue for standardizing how we share and process data in scientific workflows at the network-level to maximize step re-use and workflow portability across platforms and networks in pursuit of a foundational workflow stack. We look to evolve workflows from steps connected point-to-point in a directed acyclic graph (DAG) to steps connected via shared channels in a message system implemented as a network service. To start this evolution, we contribute: a preliminary reference model, architecture, and open tools to implement the architecture today. Our goal stands to improve the deployment and operation of complex workflows by decoupling data sharing and data processing in workflow steps. We seek the workflow community’s input on this approach’s merit, related research to explore and initial requirements from the workflows community to inform future research. I Preliminary Reference Model and Architecture Here we introduce our reference model and architecture and discuss the implications for workflows if we identify a foundational workflow stack [ 1 ] in each wide-area network (WAN) a workflow spans. Starting with changes at the workflow level, we propose a shift from modeling workflows strictly as DAGs (Fig. 1a ) to compute components connected to a messaging system [ 2 ] which may span multiple WANs (Fig. 1b ). Conceptually, the DAG remains a valid planning model, but now edges in the graph require a many-to-many data channel, or publish-subscribe (Pub/Sub) model, instead of a point-to-point Transmission Control Protocol/Internet Protocol (TCP/IP) connection. This channel permits independent scaling of bottleneck steps in a workflow (i.e. C in Fig. 1a ). It also permits effective modification of workflows. If we must update step A with A_{2} in Fig. 1b , we first bring A_{2} online, have it read M_{1} , C_{1} and C_{2} outputs from the shared channels and write the new output to a new object store prefix. Once sufficient testing passes on live data, a workflow engineer can then migrate downstream steps to A_{2} ’s object store prefix. (a) As-is workflow directed acyclic graph representation (b) To-be workflow message bus representation Fig. 1: A depiction of the current representation of a workflow in a DAG and the to-be representation separating processing and data sharing for independent scaling and scheduling. (a) Model (b) Architecture (c) Implementation Fig. 2: Proposed reference model, architecture and potential means to implement data-sharing and processing interfaces. We now discuss network-level changes to achieve the aforementioned workflow flexibility. While most large organizations endorse the Open Systems Interconnection (OSI) model for network interoperability, in reality, the internet leverages the four-layer TCP/IP model [ 3 ] . Our reference model (Fig. 2a ) suggests that complex workflows mandate open protocols and hardware implementations of the two OSI layers stuffed into applications in the TCP/IP stack: the Session and Presentation Layers. We agree with the position of others [ 4 , 2 ] that applications, or steps in a workflow, exchanging data should avoid direct sharing in favor of middleware to achieve portability, interoperability and other desirable qualities. Our model reflects this as data producers (applications) send data to the Session Layer as workflow input, then workflow steps pull inputs and push outputs to this layer until a data consumer (application) pulls the workflow output from the Session Layer for ingest (see Fig. 2a , 2b and 1b ). Below the Session Layer, TCP/IP Transport, Internet and Link Layer protocols suffice. Available open- decentralized (peer-to-peer) middleware solutions [ 4 ] warrant experimentation to provide the Session Layer service, as do various open- centralized solutions (e.g. Kafka, NiFi, MinIO, etc.). We see use cases for both, just as ad-hoc (decentralized) networks and router-based (centralized) networks exist to support network-layer connectivity. Ideally, one extends existing protocols [ 4 , 5 ] in the future to dynamically shift from peer-to-peer to a centralized service based on data quality of service (QoS) requirements, live network conditions and availability of the centralized service. As depicted in Fig. 2c , we suggest data-intensive workflows require Session-Layer devices in each network, a centralized data-distribution approach, to ease resources required from workflow input applications and individual workflow steps for data sharing downstream. We call these devices data-sharing interfaces and they are akin to routers for the Internet Layer. They decouple workflow steps from each other; only requiring a data producer to use resources to share data with the interface, not all downstream steps. We can also implement access controls for all data and enforce other data standards at these interfaces. We posit a mature architecture mandates categories of data-sharing interfaces with different performance characteristics, or QoS , in terms of max message size, latency and throughput. For example, exchanges of small messages requiring sub-second delay should employ a stream interface implemented in a stand-alone software system (e.g. Kafka, others). Data exchanges of large, infrequently-updated files certainly do not require the same QoS nor should we force them into a data stream. Therefore, a object store interface should support reading and writing of such files among workflow steps. We see both the stream and object store interfaces implementing a Pub/Sub model. For the batch interface , we envision a tool like Apache NiFi pulling data from steps running in legacy systems via a protocol that application accepts (e.g. SFTP) and pushing to a stream or object store to serve downstream steps. Also, we see a batch interface windowing streams of small events into batches written to object storage, perhaps to efficiently feed a machine-learning algorithm’s inference or training steps. Initially, we expect these interfaces implemented in WANs supporting complex workflows in an à la carte manner; only those required. Leveraging Session-Layer interfaces, in the Presentation Layer we free workflow steps from WAN -specific-data dependencies and through containerization from external software dependencies. The original OSI paper [ 6 ] states: The purpose of the Presentation Layer is to provide the set of services which may be selected by the Application Layer to enable it to interpret the meaning of the data exchanged. We argue that most workflow steps fit into this definition. Machine-learning algorithms too as they process data to enrich (i.e. classification, score, embeddings) or summarize for applications to display. This view frees most workflow steps from specific servers enabling movement across networks. Now we envision an architecture with data-processing interfaces that permit scheduling and running of containerized workflow steps, or high-performance computing (HPC) compute steps, sharing data through our data-sharing interfaces. Since we standardize data sharing across WANs for inputs and outputs, we more easily move, update, inject or delete decoupled steps in distributed workflows with minimal changes required across the entire workflow. II Conclusion We presented a network-level approach to identifying the foundational workflow stack with recommendations for standard data processing and sharing services via consistent interfaces across WANs . This approach differs from industry and academic focus on improving data operations within proprietary and/or open platforms at the Application Layer; a result of the TCP/IP model’s dominance. If these interfaces interest the workflow community, future work exists: 1) empirical testing of data-sharing interfaces in workflows, 2) supporting efficient data replication, as required, between interfaces in