A Defect Taxonomy for Infrastructure as Code: A Replication Study
Wendell Oliveira, Filipe Paiva, Thiago Emmanuel Pereira, João Brunet · 2025
Read A Defect Taxonomy for Infrastructure as Code: A Replication Study by Wendell Oliveira, Filipe Paiva, Thiago Emmanuel Pereira, João Brunet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01568.txt.
Introduction
Wendell Oliveira, Filipe Paiva, Thiago Emmanuel Pereira, and João Brunet check whether a catalog of software defects, built for one automation language, fits the newer code-based tools. Infrastructure as code scripts run modern clouds, and a shared defect taxonomy gives teams a common language for what goes wrong. The original taxonomy came only from Puppet, a declarative tool. Their 2025 replication mines 3,364 defect-related commits across two hundred eighty-five open projects using tools like Pulumi and the CDK family. They derive a taxonomy for those programming-language tools and upgrade the automatic classifier from the first study.
Replication is how a taxonomy earns trust beyond its birthplace. If the categories bend or break on new tools, the field learns where. The authors do the unglamorous work of testing a foundation. Their data set is shared for the next check. The paper serves reliability engineers who manage cloud code.
Catalog record
| CREATOR | Wendell Oliveira, Filipe Paiva, Thiago Emmanuel Pereira, João Brunet |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01568
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Defect Taxonomy for Infrastructure as Code: A Replication Study Wendell Oliveira, Filipe Paiva, Thiago Emmanuel Pereira, João Brunet Abstract Background: As Infrastructure as Code (IaC) becomes standard practice, ensuring the reliability of IaC scripts is essential. Defect taxonomies are valuable tools for this, offering a common language for issues and enabling systematic tracking. A significant prior study developed such a taxonomy, but based it exclusively on the declarative language Puppet. It remained unknown whether this taxonomy applies to programming language-based IaC (PL-IaC) tools like Pulumi, Terraform CDK, and AWS CDK. Aim: We replicated this foundational work to assess the generalizability of the taxonomy across a broader and more diverse landscape. Method: We performed qualitative analysis on 3,364 defect-related commits from 285 open-source PL-IaC repositories (PIPr dataset) to derive a PL-IaC specific defect taxonomy. We then enhanced the ACID tool, originally developed for the prior study, to automatically classify and analyze defect distributions across an expanded dataset—447 open-source repositories and 94 proprietary projects from VTEX (e-commerce) and Nubank (financial)—incorporating modern PL-IaC tools absent in the original work. Results: Our research confirmed the same eight defect categories identified in the original study, with idempotency and security defects appearing infrequently but persistently across projects. Configuration Data defects maintain high frequency in both open-source and proprietary codebases. Despite differences in project types, overall defect proportions remain similar. Conclusions: Our replication supports the generalizability of the original taxonomy, suggesting IaC development challenges surpass organizational boundaries. Configuration Data defects emerge as a persistent high-frequency problem, while idempotency and security defects remain important concerns despite lower frequency. These patterns appear consistent across open-source and proprietary projects, indicating they are fundamental to the IaC paradigm itself, transcending specific tools or project types. I Introduction Infrastructure as Code (IaC) [ 15 ] has established itself as an important practice in modern software development environments. By enabling the management of infrastructure through code, IaC has transformed how organizations deploy and maintain their computing resources. This approach has been widely adopted across industries, with tools such as Terraform [ 7 ] , Pulumi [ 22 ] , AWS CDK [ 1 ] , Ansible [ 4 ] , Chef [ 8 ] , and Puppet [ 25 ] becoming standard components in DevOps toolchains [ 12 ] . Like traditional software code, IaC scripts are susceptible to defects and quality issues. These defects can propagate throughout the infrastructure, potentially leading to system failures, security vulnerabilities, and operational inefficiencies. Understanding the nature, patterns, and impact of these defects is important for developing effective quality assurance strategies and tooling for IaC. Some important studies have examined defects in IaC scripts to shed a light on keys aspects about their characteristics, such as, causes, fixes [ 10 , 31 , 27 ] , prevalence, co-occurrence, and impact [ 5 , 6 ] . In this context, three notable works stand out for providing taxonomies and categorization frameworks. The first one established a taxonomy focused on categorizing security-related defects in IaC scripts [ 29 ] . Following this, the same research group conducted a replication study [ 30 ] , extending their framework to include Ansible and Chef scripts, thereby enhancing its applicability and relevance. Subsequently, in their work titled “Gang of Eight: A Defect Taxonomy for Infrastructure as Code Scripts” , the authors expanded their research by developing a more comprehensive defect categorization framework. This framework extends beyond security concerns to encompass a broader spectrum of defect types identified through Puppet [ 25 ] script analysis [ 28 ] . This last work in particular is the subject of our replication. The “Gang of Eight” taxonomy, while valuable, has not yet been comprehensively evaluated across emergent IaC technologies. Recently, Programming Language-based Infrastructure as Code (PL-IaC) approaches such as Terraform, Pulumi, and AWS Cloud Development Kit (AWS-CDK) have achieved significant adoption in cloud infrastructure management [ 13 , 21 ] . These tools utilize general-purpose programming languages including Python, Go and TypeScript to define infrastructure configurations. According Pulumi’s oficial reports [ 24 , 11 ] , between 2020 and 2023, the technology expanded its end user base from approximately 15,000 to 150,000—a tenfold increase. NPM downloads of Pulumi’s core packages surged fifteen-fold during this same period. This significant adoption rate creates an opportunity to evaluate how well the framework addresses the needs of modern IaC technologies. Also, most existing research has focused on open-source repositories, leaving a significant gap in our understanding of how defect patterns may vary across different implementation contexts. Hence, expanding beyond open-source to include enterprise implementations would validate the existing defect categorization framework, ensuring its continued relevance to current infrastructure practices. To bridge this gap, we replicated Rahman et al.’s study [ 28 ] to validate both the extensibility of their defect categorization framework (through a qualitative study) and their findings on the frequency and distribution of defect types (via an empirical study). Following their original methodology, we constructed our taxonomy using the PIPr dataset [ 34 ] , which contains open-source PL-IaC projects. For the empirical analysis, we drew from both the PIPr dataset and proprietary projects from VTEX 1 1 1 https://vtex.com and Nubank 2 2 2 https://www.nubank.com.br . We selected 285 repositories from the PIPr dataset based on the original study’s inclusion criteria [ 20 ] . Our proprietary sample comprised 63 projects from VTEX—a leading Latin American e-commerce platform powering over 2,000 stores in 32 countries—and 31 projects from Nubank, Brazil’s largest digital bank, serving more than 70 million customers. In total, we analyzed 94 proprietary projects. By combining PL-IaC projects from both open-source and enterprise environments, we enabled a comprehensive evaluation of the framework’s applicability across diverse technological contexts. We conducted a manual analysis of 6,612 code commits. Of these, 3,364 commits were identified as defect-related and subjected to additional manual classification. This detailed analysis allowed us to uncover a taxonomy specific to our research context, enabling comparison with the findings from the original study. Then, we evolved ACID, the tool designed in the original study to automatically categorize defects in IaC. This enhanced version handles PL-IaC files related to Terraform, Pulumi, and AWS CDK, enabling broader applicability across other modern cloud infrastructure provisioning tools. Finally, we applied our enhanced ACID version across 541 repositories of the three datasets to address the original study’s research questions regarding defect categories and their prevalence. Our comparative analysis confirmed that the original “Gang of Eight” defect taxonomy remains valid across modern PL-IaC tools and diverse environments. Configuration Data defects were consistently the most prevalent category, while Idempotency and Security defects occurred less frequently but persistently. Despite differences in project types and technologies, defect proportions were remarkably similar between the original study (15.3%) and this replication (16.8%). We propose a minor extension to the taxonomy by refining the Service category with two subcategories—Resource and Panic—to provide more detailed classification of service-related defects. Our main contributions are: • Empirical evidence regarding the generalizability of existing defect categories across different IaC technologies and implementation environments; • Comparative analysis between the original study’s findings and our results across additional technologies, open-source, and proprietary projects; and • An adaptation and evolution of ACID, a tool designed to automatically categorize defects in PL-IaC programs.