Toward Automated Test Generation for Dockerfiles Based on Analysis of Docker Image Layers DOI: XXXXXXX.XXXXXXX Conference: The 29th International Conference on Evaluation and Assessment in Software Engineering; 17–20 June, 2025; Istanbul, Turkey ISBN: 978-X-XXXX-XXXX-X/YYYY/MM CCS: Software and its
Yuki Goto, Shinsuke Matsumoto, Shinji Kusumoto · 2025
Read Toward Automated Test Generation for Dockerfiles Based on Analysis of Docker Image Layers DOI: XXXXXXX.XXXXXXX Conference: The 29th International Conference on Evaluation and Assessment in Software Engineering; 17–20 June, 2025; Istanbul, Turkey ISBN: 978-X-XXXX-XXXX-X/YYYY/MM CCS: Software and its by Yuki Goto, Shinsuke Matsumoto, Shinji Kusumoto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18150.txt.
Introduction
Yuki Goto, Shinsuke Matsumoto, and Shinji Kusumoto work on testing Docker build files in this two thousand twenty-five paper. A Dockerfile is the recipe that builds a container image. Like other code, it should be tested, but it has no branches, so classic coverage goals fail. Search-based test generators built for normal languages do not transfer. The authors propose judging tests by outputs, not by steps.
Their method analyzes the layers of built images to decide which files a test should check. It then generates tests around those results. This reframes the goal: a good test catches wrong build products. The paper is an early step toward automated Dockerfile testing. DevOps engineers and software testing researchers are the audience.
Catalog record
| CREATOR | Yuki Goto, Shinsuke Matsumoto, Shinji Kusumoto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-18150
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Toward Automated Test Generation for Dockerfiles Based on Analysis of Docker Image Layers DOI: XXXXXXX.XXXXXXX Conference: The 29th International Conference on Evaluation and Assessment in Software Engineering; 17–20 June, 2025; Istanbul, Turkey ISBN: 978-X-XXXX-XXXX-X/YYYY/MM CCS: Software and its engineering CCS: Software and its engineering Software testing and debugging Yuki Goto, Shinsuke Matsumoto, Shinji Kusumoto Abstract . Docker has gained attention as a lightweight container-based virtualization platform. The process for building a Docker image is defined in a text file called a Dockerfile. A Dockerfile can be considered as a kind of source code that contains instructions on how to build a Docker image. Its behavior should be verified through testing, as is done for source code in a general programming language. For source code in languages such as Java, search-based test generation techniques have been proposed. However, existing automated test generation techniques cannot be applied to Dockerfiles. Since a Dockerfile does not contain branches, the coverage metric, typically used as an objective function in existing methods, becomes meaningless. In this study, we propose an automated test generation method for Dockerfiles based on processing results rather than processing steps. The proposed method determines which files should be tested and generates the corresponding tests based on an analysis of Dockerfile instructions and Docker image layers. The experimental results show that the proposed method can reproduce over 80% of the tests created by developers. 1. Introduction Docker has gained attention as a lightweight container-based virtualization platform. Container-based virtualization enables the execution of applications without additional software installation in the local environment. This approach not only enhances reproducibility and portability but also enables rapid deployment ( Sharma et al. 2016 ) . Due to these advantages, Docker is widely used in software development ( Wu et al. 2023 ) . Moreover, it has been leveraged as a way to ensure reproducibility in academic research ( Boettiger 2015 ) . The process for building a Docker image is defined in a text file called a Dockerfile. A Docker image is built from this Dockerfile and a virtualized environment called a container is built from the Docker image. A Dockerfile can be considered as a kind of source code that contains instructions on how to build a Docker image. The behavior of Dockerfiles should be verified through testing, as is done for source code in a general programming language. In Dockerfiles, instructions that require network communication are frequently executed. These include package installation instructions via package managers and downloads using the wget or curl command. Consequently, system degradation is more likely to occur due to external factors rather than the contents of a Dockerfile itself ( Henkel et al. 2021 ) . Henkel et al. found that 26% of the examined Dockerfiles on GitHub failed to build ( Henkel et al. 2021 ) . Most of these failures were caused by changes in external environments, such as changes in dependency resolution. Creating tests for a Dockerfile can also be useful for preserving behavior during Dockerfile refactoring. Numerous automated test generation techniques have been proposed to support the testing of software written in a general programming language ( Anand et al. 2013 ; McMinn 2004 ) . Automatically generated tests are used for degradation prevention and refactoring. Test generation techniques involve providing various input values for the arguments of methods and focusing on execution paths to maximize coverage. For instance, EvoSuite ( Fraser and Arcuri 2011 ) , a search-based test generation tool for Java, is commonly used. EvoSuite can generate tests with high coverage. Fraser and Arcuri reported that EvoSuite achieved 71% branch coverage per class in large-scale empirical experiments ( Fraser and Arcuri 2014 ) . Test generation for Dockerfiles could also provide developer support. However, existing search-based automated test generation techniques cannot be applied to Dockerfiles because a Dockerfile does not contain branches, only a single execution path. As a result, the coverage metric, which is typically used as an objective function in existing search-based testing, becomes meaningless. Therefore, the fundamental concept of exploratory testing cannot be applied. A different approach is thus required. In this study, we propose an automated test generation method for Dockerfiles. The key concept of the proposed method is to generate tests based on processing results, rather than processing steps. A Docker image built from a Dockerfile is considered as a set of effects produced by the Dockerfile. Based on an analysis of Dockerfile instructions and Docker image layers, it is determined which files should be tested and the corresponding tests are generated. Experimental evaluation confirms that the proposed method can reproduce over 80% of the tests created by developers. 2. Preliminaries 2.1. Dockerfile and Layers A Dockerfile is a text file that describes the steps required to build a Docker image. fig. 1(a) shows an example of a Dockerfile that contains instructions to set up a Python runtime environment and execute main.py . First, the FROM instruction specifies the base image, which serves as the foundation of the Docker image. Next, the RUN instruction executes shell commands to install Python using a package manager. Then, the COPY instruction copies the local main.py file into the image. Finally, the CMD instruction sets the shell command to be executed when the container starts. As shown in fig. 1(a) , multiple shell commands can be executed in the RUN section by using the shell operator && . fig. 1(b) shows the Docker image built from the Dockerfile in fig. 1(a) . A Docker image can be considered as a snapshot of the configured environment. A container, which is a virtual environment, is created from a Docker image. (a) Example of Dockerfile ⬇ # Set the base image FROM debian : latest # Install Python RUN apt - get update && \ apt - get install - y \ python3 # Copy main.py COPY main . py . # Execute the script # when starting a container CMD [ "python3" , "main.py" ] (b) Built Docker image Figure 1 . Build execution example A Docker image is composed of layers. Each layer corresponds to an instruction in a Dockerfile. During the build process, Dockerfile instructions are executed sequentially from top to bottom. Each instruction generates a new layer that stores added, changed, and removed files. For the Dockerfile shown in fig. 1(a) , the build process begins with the execution of the FROM instruction. As shown in fig. 1(b) , the layer created by the FROM instruction contains a large number of files required by the base image. Next, when the RUN instruction is executed to install Python, a new layer is added. This includes the python3 binary along with related files and cached files generated by apt-get . Following this, the COPY instruction creates an additional layer, where the main.py file is added. Finally, the CMD instruction sets metadata for the image without changing the filesystem, so no additional layer is created. These layers are transparently overlaid to form a unified file system. 2.2. Container Structure Tests Container Structure Tests (CST) 1 1 1 https://github.com/GoogleContainerTools/container-structure-test is a testing framework for Dockerfiles. It can be used to verify the output of shell commands in a container and the existence of files. The