Can a domain-specific language improve program structure comprehension of data pipelines? A mixed-methods study.

Philip Heltweg, Georg-Daniel Schwarz, Dirk Riehle · 2025

Read Can a domain-specific language improve program structure comprehension of data pipelines? A mixed-methods study. by Philip Heltweg, Georg-Daniel Schwarz, Dirk Riehle (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16764.txt.

Introduction

Philip Heltweg, Georg-Daniel Schwarz, and Dirk Riehle test whether a language built for one job helps people grasp the shape of data pipeline code. Such a tongue should let field experts join shared projects, but only if they can read code that others wrote.

In a trial, people using the special language took the same time as a control group, but they got far more answers right. In a follow-up survey, users praised the clear overview of the pipeline, the firmer shape the language enforces, and its closeness to how they picture a pipeline. Less coding skill was needed, and past tool use carried over.

Catalog record

CREATORPhilip Heltweg, Georg-Daniel Schwarz, Dirk Riehle
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16764

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can a domain-specific language improve program structure comprehension of data pipelines? A mixed-methods study. Philip Heltweg, Georg-Daniel Schwarz, Dirk Riehle Abstract In many application domains, domain-specific languages can allow domain experts to contribute to collaborative projects more correctly and efficiently. To do so, they must be able to understand program structure from reading existing source code. With high-quality data becoming an increasingly important resource, the creation of data pipelines is an important application domain for domain-specific languages. We execute a mixed-method study consisting of a controlled experiment and a follow-up descriptive survey among the participants to understand the effects of a domain-specific language on bottom-up program understanding and generate hypotheses for future research. During the experiment, participants need the same time to solve program structure comprehension tasks, but are significantly more correct when using the domain-specific language. In the descriptive survey, participants describe reasons related to the programming language itself, such as a better pipeline overview, more enforced code structure, and a closer alignment to the mental model of a data pipeline. In addition, human factors such as less required programming experience and the ability to reuse experience from other data engineering tools are discussed. Based on these results, domain-specific languages are a promising tool for creating data pipelines that can increase correct understanding of program structure and lower barriers to entry for domain experts. Open questions exist to make more informed implementation decisions for domain-specific languages for data pipelines in the future. Preprint Keywords program comprehension \cdot data pipelines \cdot data engineering \cdot domain-specific languages \cdot mixed-methods study \cdot open data 1 Introduction Domain-specific languages (DSLs) can be a useful alternative to general-purpose programming languages (GPLs) in many application domains. By focussing on one domain, they can have a reduced scope and re-use glossary and concepts from the application domain, making them easier to learn and more efficient to program for domain experts ( Kosar et al.,, 2018 ; Johanson and Hasselbring,, 2017 ) . However, because DSLs are a specialized tool, they have to be carefully evaluated to determine whether they provide enough benefits to make their adoption a good choice. When working on non-trivial software applications, developers must first understand the program structure from source code. Only then can they make changes to extend existing implementations or fix bugs. Program comprehension, in general, is estimated to be the dominant activity while programming, with more than 50% of time spent ( Roberto Minelli and Lanza,, 2015 ; Xia et al.,, 2018 ) . Therefore, the effects of a DSL on program structure comprehension are essential for the usefulness of a DSL in an application domain. The evaluation of DSLs generally has to be domain-specific ( Kosar et al.,, 2018 ) . Increasingly, high-quality data, and with it data engineering, is of large importance in industry because many innovative apps and AI applications rely on access to data. Sources for data sets vary from company internal data to open data, with open data mainly published by governments but also by some private entities. Depending on the type of data, creating an automated data pipeline is a major part of data engineering. An example is regularly changing data, such as schedules released as open transport data, that should be ingested and improved automatically with updated releases. In complex domains, data-engineers must collaborate with subject-matter experts to understand the meaning of data. A common challenge during these collaborations is that subject-matter experts lack programming experience, which complicates it to find a shared collaboration artifact with professional programmers ( Heltweg and Riehle,, 2023 ) . Domain-specific languages can be a useful middle-ground, that enables subject-matter experts to contribute directly to the creation of data pipelines, as previously shown in other domains ( Johanson and Hasselbring,, 2017 ; Lopes et al.,, 2021 ) . DSLs can be grounded in the formal and informal glossary of domain experts, such as sketches ( Wile,, 2004 ) . A common mental model for a data pipeline is a graph of processing steps connected by pipes, known from visual programming. A DSL can provide an explicit syntax and semantics to express this data pipeline structure with the pipes and filters architecture. In previous explorative work, we found using a domain-specific language based on this architecture had positive effects on speed, quality of the solution and perceived difficulty when solving data engineering exercises on real life open data sets ( Heltweg et al.,, 2025 ) . Building on this high-level validation, we aim to understand how domain-specific languages contribute to improved performance by subject-matter experts and what language features are important in more detail. To do so, we conduct a series of empirical evaluations using quantitative and qualitative methods. Previous research shows that programming language research lacks empirical studies, instead focusing on solution proposals ( do Nascimento et al.,, 2012 ) . However, empirical user studies to evaluate usability are essential tools that can lead to insights that would not have been gained otherwise ( Buse et al.,, 2011 ; Barišić et al.,, 2018 ) . When contributing to a collaborative data engineering project, the first thing a subject-matter expert will need to do is read and understand the intention behind data pipeline source code. To start, we therefore focus on bottom-up program comprehension, the process of inferring the intentions behind an implementation from reading source code ( Wyrich et al.,, 2023 ) ; we do so in the domain of building data pipelines by non-professional programmers (subject-matter experts). In the context of this mixed-methods study, we compare data pipelines implemented in a DSL using an explicit pipes and filters architecture (Jayvee) to imperative scripts in a GPL with libraries for data engineering (Python with Pandas). We performed an initial controlled experiment to gather quantitative data on task performance in terms of time and correctness. In a follow-up survey, we look for causal influences for the experiment outcomes.. With the results, we answer the following research questions: Research Question 1: Do data pipelines implemented in Jayvee change bottom-up program structure comprehension compared to Python/Pandas for non-professional programmers… a: regarding speed? b: regarding correctness? c: regarding the perceived difficulty? Research Question 2: What reasons exist for effects on program comprehension for data pipelines implemented in Jayvee compared to Python/Pandas for non-professional programmers? In this article, we contribute: 1. A mixed-methods approach, combining a controlled experiment with a descriptive survey, to evaluate the effects of DSLs in the domain of data pipeline modelling. 2. Quantitative data, based on a controlled experiment, on how strongly the use of a DSL in the domain of data pipeline modelling can influence pipeline structure understanding, contributing to the growing literature on domain-specific languages and motivating their use in data engineering. 3. Explanations for these effects from participant surveys to guide future practitioners or researchers that implement DSLs for data engineering. 2 Related Work Empirical research into the effects of domain-specific languages has been performed across multiple domains. Kosar et al. have used controlled experiments to compare DSLs with GPLs and libraries. Initially, in the context of GUI programming, they compared the DSL XAML with C# Forms, with XAML performing better for answering questions on provided source code ( Kosar et al.,, 2010 ) .

Read the complete text