Continuous Integration Practices in Machine Learning Projects: The Practitioners’ Perspective
João Helis Bernardo, Daniel Alencar da Costa, Filipe Roseiro Cogo, Sérgio Queiróz de Medeiros, Uirá Kulesza · 2025
Read Continuous Integration Practices in Machine Learning Projects: The Practitioners’ Perspective by João Helis Bernardo, Daniel Alencar da Costa, Filipe Roseiro Cogo, Sérgio Queiróz de Medeiros, Uirá Kulesza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17378.txt.
Introduction
João Helis Bernardo and co-authors survey developers about continuous integration in machine learning projects, in a 2025 paper. Continuous integration, the habit of merging and testing code often, works well for normal software but strains in ML projects, which show longer builds and thinner test coverage. The team asked 155 practitioners from 47 projects why.
Eight key differences surfaced, from test complexity to infrastructure needs. Model training, heavy data handling, and thirst for compute stretch build times, while the randomness inside ML systems makes tests flaky and hard to trust. The authors propose ML-specific tweaks to standard practice. Their message: keep the core principles, but adapt the plumbing.
Catalog record
| CREATOR | João Helis Bernardo, Daniel Alencar da Costa, Filipe Roseiro Cogo, Sérgio Queiróz de Medeiros, Uirá Kulesza |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 127 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17378
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Continuous Integration Practices in Machine Learning Projects: The Practitioners’ Perspective João Helis Bernardo, Daniel Alencar da Costa, Filipe Roseiro Cogo, Sérgio Queiróz de Medeiros, Uirá Kulesza Abstract Continuous Integration (CI) is a cornerstone of modern software development, promoting stability and ensuring quality. However, while widely adopted in traditional software projects, applying CI practices to Machine Learning (ML) projects presents distinctive characteristics. For example, our previous work revealed that ML projects often experience longer build durations and lower test coverage rates compared to their non-ML counterparts. Building on these quantitative findings, this work investigates the underlying reasons for these distinctive characteristics through a qualitative perspective. We surveyed 155 practitioners from 47 ML projects and conducted a thematic analysis to identify the key differences in CI adoption in the ML domain compared to non-ML domains. Practitioners highlighted eight key differences, including test complexity, infrastructure requirements, and build duration and stability. Common challenges mentioned by practitioners include higher project complexity, model training demands, extensive data handling, increased computational resource needs, and dependency management, all contributing to extended build durations. Furthermore, ML systems’ non-deterministic nature, data dependencies, and computational constraints were identified as significant barriers to effective testing. The key takeaway from this study is that while foundational CI principles remain valuable, ML projects require tailored approaches to address their unique challenges. To bridge this gap, we propose a set of ML-specific CI practices, including tracking model performance metrics and prioritizing test execution within CI pipelines. Additionally, our findings highlight the importance of fostering interdisciplinary collaboration to strengthen the testing culture in ML projects. Furthermore, we emphasize the need for standardized guidelines to address key CI challenges in ML workflows, such as dependency management. By bridging quantitative findings with practitioners’ insights, this study provides a deeper understanding of the interplay between CI practices and the unique demands of ML projects, laying the groundwork for more efficient and robust CI strategies in this domain. ∎ MSC 68N01 68T05 68M15 1 Introduction Machine Learning (ML) has become a cornerstone of modern software systems, driving advancements across diverse domains ( Washizaki et al. 2019 ; Gonzalez et al. 2020 ; Pallathadka et al. 2023 ) . In this paper, we define ML projects as software initiatives that integrate ML components to enable intelligent functionalities. These projects either address domain-specific problems (i.e., ML applications or ML-enabled systems) or provide general-purpose solutions (i.e., ML tools). As ML projects gain prominence and grow in complexity, ensuring their quality and reliability becomes increasingly challenging. These projects often involve computationally intensive tasks and exhibit inherent non-deterministic behavior ( Nascimento et al. 2020 ; Giray 2021 ) , making it difficult to guarantee consistent performance and reproducibility. Furthermore, ML projects face the unique challenge of deploying models to production, a task that can further strain development workflows ( Symeonidis et al. 2022 ) . To tackle these complexities, the field of Software Engineering for Machine Learning ( SE4ML ) has emerged, adapting traditional software engineering practices to the specific requirements of ML projects. Key practices that are well established in traditional software development, such as those related to DevOps ( Leite et al. 2019 ) , have been adapted to the ML domain (e.g., MLOps ( Mäkinen et al. 2021 ) ). In particular,Continuous Integration (CI) practices have become pivotal for enhancing the development efficiency and release stability of ML projects ( Gift and Deza 2021 ) . CI is a widely adopted practice in software development that involves frequent and automated integration of code changes into a shared repository, at least daily ( Fowler 2006 ; Duvall et al. 2007 ) . In the ML domain, Rzig et al. 2022 conducted an analysis revealing that approximately 37% of ML projects have integrated a CI service into their workflows. The fundamental philosophy behind CI is to ensure that the software passes all tests and can build at all times ( Duvall et al. 2007 ) . By fostering frequent commits, shorter build durations, and high test coverage, CI enhances development workflows and project reliability ( Soares et al. 2022 ) . However, while extensively studied in traditional software projects ( Vasilescu et al. 2015 ; Hilton et al. 2016 ; Zhao et al. 2017 ; Bernardo et al. 2018 ; Nery et al. 2019 ; Santos et al. 2022 ; Saraiva et al. 2023 ) , the integration of CI into ML projects remains relatively unexplored. Unlike traditional CI for non-ML projects, CI in ML environments needs to handle the inherently probabilistic nature of ML components ( Renggli et al. 2019 ) . As such, in the context of ML, CI extends beyond merely testing and validating code and components and also involves testing and validation of data, data schemas, and models ( Karamitsos et al. 2020 ) . Our previous work conducted a quantitative analysis of 93 ML projects and 92 non-ML projects, examining the differences in the adoption of four key CI practices (i.e., frequent code integration, short build duration, quick build fixes, and comprehensive test coverage) between these two types of projects. Our findings indicated that ML projects often require longer build durations, while some exhibit lower test coverage compared to non-ML projects ( Bernardo et al. 2024 ) . While these insights are valuable, the underlying reasons for the observed differences in build duration remain unclear. A deeper understanding of the specific barriers faced by ML projects in adopting CI is crucial to inform practitioners and enable the full potential of CI in improving code quality and overall project success. The general goal of this study is to deepen the understanding of the perceptions of ML practitioners regarding the differences, specific barriers, and strategies associated with CI adoption in ML projects. Through thematic analysis of survey responses from 155 ML practitioners, we identify key themes – such as computational complexity, dependency management, and data handling – that contribute to extended build durations and reduced test coverage in ML projects. These findings provide actionable insights for improving CI in the ML domain, fostering more efficient CI workflows, and advancing the integration of CI practices in this critical area of software development. By bridging existing quantitative findings with practitioners’ experiences, this study offers valuable guidance for researchers and practitioners aiming to optimize CI workflows for ML projects. Our investigation is guided by the following Research Questions (RQs): • RQ1: What are the perceived reasons behind the differences between ML and non-ML projects with respect to their CI practices? Motivation: Our prior work ( Bernardo et al. 2024 ) identified significant differences in CI practices between ML and non-ML projects. However, the reasons behind these differences remain unclear. Gaining insights from ML practitioners’ perspectives can shed light on the unique challenges of CI adoption in ML projects and guide the development of tailored solutions. Findings: Participants of our survey identified key differences in CI practices between ML and non-ML projects, particularly in test coverage (61.9%) and build duration (63.2%), while perceptions regarding fixing broken builds and commit frequency were more neutral. Challenges in CI implementation mentioned by the participants were categorized into eight main themes, including testing complexity, infrastructure requirements, and build duration stability. Co-occurrence analysis revealed strong interdependencies among these challenges,