A Study on Messaging Trade-offs in Data Streaming for Scientific Workflows

Anjus George, Michael J. Brim, Christopher Zimmer, Tyler J. Skluzacek, A.J. Ruckman, Gustav R. Jansen, Sarp Oral · 2025

Read A Study on Messaging Trade-offs in Data Streaming for Scientific Workflows by Anjus George, Michael J. Brim, Christopher Zimmer, Tyler J. Skluzacek, A.J. Ruckman, Gustav R. Jansen, Sarp Oral (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07199.txt.

Description

Big experiments now stream data straight from instruments to supercomputers, skipping slow file transfers. Anjus George and 6 co-authors study the messaging settings behind that streaming in a 2025 research paper. They work with RabbitMQ, a widely used messaging framework, inside the data streaming infrastructure at the Oak Ridge Leadership Computing Facility. Two real scientific workflows guide their tests.

The team uses synthetic workloads modeled on the Deleria and LCLS workflows. It varies delivery settings and measures throughput, the amount of data moved per second. They focus on the cost of reliable delivery, where every message must arrive intact. Their simulations surface practical trade-offs between speed and reliability. The findings help workflow teams pick configurations that keep analysis fast enough to steer experiments while they run.

Catalog record

CREATORAnjus George, Michael J. Brim, Christopher Zimmer, Tyler J. Skluzacek, A.J. Ruckman, Gustav R. Jansen, Sarp Oral
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-07199

Browse the full library · About the Tower of Babel