Deployability-Centric Infrastructure-as-Code Generation: An LLM-based Iterative Framework
Tianyi Zhang a † \dagger , Shidong Pan b,c † \dagger , Zejun Zhang d , Zhenchang Xing a,e , Xiaoyu Sun a * · 2025
Read Deployability-Centric Infrastructure-as-Code Generation: An LLM-based Iterative Framework by Tianyi Zhang a † \dagger , Shidong Pan b,c † \dagger , Zejun Zhang d , Zhenchang Xing a,e , Xiaoyu Sun a * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05623.txt.
Introduction
Tianyi Zhang and co-authors test whether language models can write cloud setup code that truly deploys in a 2025 paper. Such templates are often judged on looks, yet real worth is whether they run. Top models passed that bar on only a fifth to a third of first tries.
A new benchmark grades more than one hundred fifty real scenarios across many services. Their framework, IaCGen, loops through format, syntax, and live deploy checks, fixing errors each round. Within ten rounds most templates became deployable. Direct human hints on deploy errors lifted results further. The work mirrors how DevOps teams ship in practice.
Catalog record
| CREATOR | Tianyi Zhang a † \dagger , Shidong Pan b,c † \dagger , Zejun Zhang d , Zhenchang Xing a,e , Xiaoyu Sun a * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05623
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deployability-Centric Infrastructure-as-Code Generation: An LLM-based Iterative Framework Tianyi Zhang a † \dagger , Shidong Pan b,c † \dagger , Zejun Zhang d , Zhenchang Xing a,e , Xiaoyu Sun a * Abstract Infrastructure-as-Code (IaC) generation holds significant promise for automating cloud infrastructure provisioning. Recent advances in Large Language Models (LLMs) present a promising opportunity to democratize IaC development by generating deployable infrastructure templates from natural language descriptions, but current evaluation focuses on syntactic correctness while ignoring deployability, the fatal measure of IaC template utility. We address this gap through two contributions: (1) IaCGen, an LLM-based deployability-centric framework that uses iterative feedback mechanism to generate IaC templates, and (2) DPIaC-Eval, a deployability-centric IaC template benchmark consists of 153 real-world scenarios that can evaluate syntax, deployment, user intent, and security. Our evaluation reveals that state-of-the-art LLMs initially performed poorly, with Claude-3.5 and Claude-3.7 achieving only 30.2% and 26.8% deployment success on the first attempt respectively. However, IaCGen transforms this performance dramatically: all evaluated models reach > > 90% passItr@25, with Claude-3.5 and Claude-3.7 achieving 98% success rate. Despite these improvements, critical challenges remain in user intent alignment (25.2% accuracy) and security compliance (8.4% pass rate), highlighting areas requiring continued research. Our work provides the first comprehensive assessment of deployability-centric IaC template generation and establishes a foundation for future research. $\dagger$ $\dagger$ footnotetext: Equal contribution. * * footnotetext: Corresponding author. [email protected]. I Introduction The digital transformation of enterprises has fundamentally altered how organizations manage and deploy computing infrastructure. As cloud adoption accelerates globally, with over 82% of enterprises now utilizing cloud services [ 1 ] , the complexity and scale of infrastructure management have increased exponentially. Infrastructure-as-Code (IaC) has emerged as the default standard for managing this complexity, enabling organizations to define, provision, and maintain cloud resources through programmable templates that ensure consistency, repeatability, and version control across deployments [ 2 , 3 , 4 ] . This paradigm shift from manual infrastructure management to code-driven automation has become essential for modern DevOps practices [ 5 ] , supporting the rapid deployment cycles and scalability requirements of contemporary software systems. However, the widespread adoption of IaC faces significant barriers rooted in its inherent technical complexity and the specialized expertise it demands [ 6 , 7 , 8 ] . Creating effective IaC templates requires a deep understanding of cloud service architectures, resource interdependencies, and platform-specific constraints that extend far beyond traditional programming knowledge [ 5 ] . The steep learning curve of IaC languages creates substantial obstacles for development teams lacking dedicated DevOps expertise [ 9 , 10 ] . This expertise gap not only slows development speed but also increases the risk of misconfigurations that can lead to security vulnerabilities and service outages [ 11 , 12 ] . Recent advances in Large Language Models (LLMs) present a promising opportunity to democratize IaC development by generating deployable infrastructure templates from natural language descriptions [ 13 , 14 , 15 ] . An effective LLM-based IaC generator could potentially transform infrastructure automation by enabling developers to describe their infrastructure requirements in natural language and receive working deployable IaC in return. This capability would significantly accelerate development workflows and reduce the expertise barrier for cloud adoption. Despite the promising potential of LLMs in IaC automation, existing research suffers from a critical limitation: an overemphasis on syntactic correctness while neglecting the fundamental requirement of deployability. In operational contexts, deployability represents the primary measure of IaC template quality, as non-deployable templates provide no practical value regardless of syntactic correctness. Furthermore, previous research indicates that LLMs demonstrate considerably lower performance in IaC generation compared to general programming tasks [ 9 , 16 ] , with initial success rates as low as 19%. To mitigate research gaps, we present DPIaC-Eval, a novel benchmark comprising 153 real-world infrastructure tasks across 58 unique AWS services, with each task including a verified deployable IaC template as ground truth. The formative study shows that mainstream LLMs can only achieve a merely 50% success rate in the simplest scenarios. Therefore, we propose IaCGen, a novel framework that consists of two modules, Iterative IaC Generation and IaC Deployment, integrated through an iterative feedback mechanism that mirrors real DevOps workflows. This design enables models to learn from syntactic and deployment failures, progressively refining their templates. To evaluate IaCGen, we first assess the effectiveness of IaC template generation with six popular LLMs. The results show that using LLM alone achieves only 20.8 \sim 30.2% correctness, while IaCGen achieves 39.0% \sim 66.7%, with 54.6% \sim 91.6% correctness within 5 and 10 iterations. Since LLMs exhibit performance differences, we further identify five error patterns and find that different models tend to make different types of error, with the Claude models showing the best performance in avoiding errors. Additionally, we evaluate whether IaCGen can achieve better performance on generating IaC templates with human-in-the-loop feedback. We find that human feedback can improve overall correctness by over 19.5%, help all six LLMs achieve more than 90% passItr@25 accuracy. To verify the trustworthiness of the generated IaC templates, we evaluate their user intent matching with resources and attributes from ground truth, and check the security compliance by Checkov. Over 40% of the templates satisfy resource or attribute requirements, and over 70% meet security compliance by security policies. Furthermore, we discuss the generalizability of IaCGen, its implications for SE-inspired LLM-based frameworks, and the emerging research directions concerning vulnerabilities in IaC templates. In summary, our paper makes the following contributions: • DPIaC-Eval, the first deployability-centric benchmark for IaC template generation, containing 153 real-world IaC tasks across five difficulty levels and 58 AWS services, with multi-dimensional validation spanning syntax, deployment, user intent, and security. • IaCGen, an LLM-based framework on IaC template generation with iterative feedback mechanism, achieving the state-of-the-art performance. • Empirical evidences about model performance across multiple dimensions of IaC template quality, providing insights into common failure patterns and priority areas for improvement. The replication package is available at https://github.com/Tianyi2/IaCGen . II Background II-A Infrastructure-as-Code IaC represents a paradigm for managing cloud infrastructure through declarative configuration files, enabling automated provisioning and management of cloud resources. The terms “IaC”, “IaC language”, and “IaC tool” are used interchangeably in the literature to refer to the domain-specific languages designed for infrastructure specification and deployment. Prominent examples include AWS CloudFormation, HashiCorp Terraform, and Azure Resource Manager. Each IaC language provides its own syntax and capabilities for defining infrastructure configurations. IaC templates are the configuration files that contain the infrastructure specifications written in a particular IaC language. These templates are typically expressed in declarative formats such as YAML, JSON, or HashiCorp Configuration Language, depending on the target IaC language. IaC templates are structured around two fundamental components (resource and parameter). Resources represent the cloud services and infrastructure components to be provisioned. The term “resources” is used instead of “services” because a single cloud service may expose multiple resource types. For instance, Amazon SNS, a message delivery service, provides four distinct resource types (Topic, Subscription, TopicPolicy, and TopicInlinePolicy). In AWS CloudFormation, resources follow the hierarchical naming convention ‘AWS::ServiceName::ResourceType’, clearly delineating the service provider, service name, and specific resource type. Parameters enable template customization by allowing external specification of resource property values. Parameters improve template reusability and flexibility by enabling