Position: More Rigorous Software Engineering Would Improve Reproducibility in Machine Learning Research Moritz Wolter, Lokesh Veeramacheneni Abstract Experimental verification and falsification of scholarly work are part of the scientific method’s core. To improve the Machine Learning-communities’ ability to verify results from prior work, we argue for more robust software engineering. We estimate the adoption of common engineering best practices by examining repository links from all recently accepted International Conference on Machine Learning (ICML), International Conference on Learning Representations (ICLR) and Conference on Neural Information Processing Systems (NeurIPS) papers as well as International Conference on Machine Learning (ICML) papers over time. Based on the results, we recommend how we, as a community, can improve reproducibility in Machine Learning (ML)-research. † † affiliationnotice: Equal contribution 1 Introduction Modern science relies on trust, that we partially establish by experimental evidence. Scientists often propose logically coherent hypotheses, then verify them experimentally ( Popper 2005 ) . Next to theoretically proven discovery. A large part of Machine Learning-research rests upon empirical foundations. Consequently, reproducibility is a key concern in Machine Learning research. When we fail to enable reproduction by other scientists, this pillar of our scientific structure becomes shaky. This position paper focuses on what we as a community are already doing to allow the reproducibility of our work and where we can still improve. Since Machine Learning is a field that is heavily reliant on software, we draw upon established software engineering best practices, measure their adoption and argue for more rigorous software engineering in Machine Learning work. Figure 1: The state of software engineering at major 2024 Machine Learning-conferences. The chart illustrates web-crawled percentages of files and folders which are tied to the adoption of software engineering best practices. Figure 1 illustrates potential avenues for improvement. The figure illustrates best practice adoption estimates. We argue that supplementary code facilitates reproduction if it comes with a license, contains a readme with reproduction steps, documents software requirements, and is carefully tested. Figure 1 illustrates our community’s potential for future growth along many of these dimensions. 2 Related work Previous work has identified missing code or data as an important obstacle to reproducibility in Machine Learning ( Hutson 2018 ) . More recently, neighbouring computational fields have started to identify software quality as a second key hindrance ( Hoyt et al. 2023 ) . In response, the Public Library of Science has proposed a set of guidelines, encouraging its authors to follow ( Pérez-Riverol et al. 2016 ; Prlic & Procter 2012 ; Sandve et al. 2013 ; List et al. 2017 ) . Within the machine learning community, the Conference on Neural Information Processing Systems has also started to address this issue by proposing its own guidelines ( Stojnic et al. 2020 ) . We will now consider elements of scientific software engineering, with a focus on standardized features, which we can measure. 2.1 Licensing software Providing a link to a public open-source project is not the same as supplying a license. Without a license, the host platform’s terms of service apply ( opensourceguide 2025 ) . On GitHub, for example, the terms allow the public to view and fork the project. Other use is not allowed ( opensourceguide 2025 ) . Therefore, providing a license is a necessary prerequisite. It allows future scientists to build on the existing codebase. ( Pérez-Riverol et al. 2016 ; opensourceguide 2025 ) . 2.2 Onboarding new users with a README Typically, ML repositories do not come with complete project documentation. Providing a README is a first step towards documentation and is considered a good practice. The README should come with a project description, a quick start guide, and share information on how to contribute ( Stojnic et al. 2020 ; Pérez-Riverol et al. 2016 ) . Providing LICENCE and README files is a language-agnostic step. We will now focus on the specifics of Python. 2.3 Python’s dominance in Machine Learning Python dominates in the repositories we found in International Conference on Machine Learning-papers (Figure 3 ). Since major Machine Learning-frameworks like PyTorch ( Paszke et al. 2017 ) , Jax ( Bradbury et al. 2025 ) or Tensorflow ( Abadi et al. 2015 ) all provide Python functionality, Python’s dominance does not surprise us. Additionally, the Conference on Neural Information Processing Systems tips for releasing code ( Stojnic et al. 2020 ) , for example, are written with Python in mind. 2.4 Sharing requirements in Python A recommended way to share requirements is the inclusion of a requirements.txt file ( Pip-developers 2025 ; Stojnic et al. 2020 ) . The file lists the code library names a project depends on with optional version numbers. Even though other file names are possible ( Pip-developers 2025 ) , most code-authors use requirements.txt as the recommended file name. Providing the file should allow others to install all requirements automatically with the package installer for Python. 2.5 Freezing conda Projects that use conda ( Conda-developers 2025 ) for dependecy management should provide a environment.yml file ( Stojnic et al. 2020 ) . The file will allow Conda to recreate the software environment in which experiments were conducted automatically. In contrast to package installer for Python, conda does not rely on a system compiler and delivers binary dependencies directly. 2.6 Tests in Python Automated tests allow users, as well as code authors, to verify that the code works as intended. This is important for new users to get started. Here, successful test runs indicate that everything is set up correctly ( List et al. 2017 ) . During development, automated tests facilitate code verification. Primarily, they allow us to check new ideas before integrating these into the core codebase ( Pérez-Riverol et al. 2016 ) . Last but not least, the test documents core development assumptions and the key behaviour of the code. When other groups or new people in the same group join a project, the tests help them ensure older features remain operational as new functionality is added. In the python-world pytest ( Pytest-developers 2025 ) provides a functional and unittests ( Python-developers 2025 ) a more object-oriented approach to testing. Regarding test organization, the Python Packaging Authority recommends a tests folder ( Python-Packaging-Authority 2025 ) , which contains the test files. The tests folder should be in the root directory of the project. Within the Jax ecosystem, Chex ( ecosystem authors 2025 ) offers a very elegant way to automatically evaluate individual tests in different Machine Learning-specific settings. Its variants decorator automatically tests a function’s compatibility with just-in-time compilation, device mapping and parallelization. 2.7 Packaging Python code Testing ensures quality and reproducibility within the original code-base. However, most community members are working on their own projects and would reuse code-modules or individual functions from others. Copying code from the original repository is a simple way to reuse code. Unfortunately, this approach leads to code duplication within the community, with all the problems that come with it ( Pérez-Riverol et al. 2016 ) . A prominent problem, for example, is that once a bug is identified, it must be manually fixed for every duplication. To combat duplication, we require a system that makes it simple to install external dependencies as a library. This is where packaging comes in. A packaged Python project can be automatically installed via package installer for Python. The process makes it available without introducing duplication through Python imports. The process is straightforward, a packaged project should have src and tests folders ( Python-Packaging-Authority 2025 ) , additionally package installer for Python needs to know about requirements and other project metadata. Such information is typically provided by a pyproject.toml file. When authors use setuptools ( Setuptools-Team 2025 ) to package their code, a setup.py or setup.cfg will appear. 2.8 Testing packaged code Packaging allows easy installation of code. To guarantee that results are reproducible we can modify the test setup to test our code and our dependencies-specification. We argue that package testing is desirable from a reproducibility point of view, because it allows authors to test their code package from a user perspective. The test process requires a tool that creates a virtual environment, installs the package, and runs the tests. Common choices in the python world are tox ( tox devs 2025 ) or nox ( Flowers et al. 2025 ) . Using tox requires us to set up a tox.toml , or tox.ini file. For nox we need a noxfile.py . Both files should appear in the root directory of the project. 2.9 Fixing seeds We require pseudorandom initialization to initialize network matrices before we begin experimenting. Neural network optimization is typically not a convex problem. In other words, there is no guarantee that the optimizer will find the global optimum from any starting point. In the interest of reproducibility, we must record the seed value we feed into the random number generator. After all, the seed value determines the starting point of the optimization process. Jax ( Bradbury et al. 2025 ) makes the state of the pseudorandom number generator explicit by introducing a unique key object. To generate keys, programmers must provide seed values by default. Their approach solves a lot of reproducibility issues. PyTorch users can optionally choose to set seed values ( PyTorch-Contributors 2024 ) . In the interest of reproducibility, this position paper argues we should make that a habit in all our scientific work. Overall, the related work suggests, use versioning for code and data ( Sandve et al. 2013 ; Stojnic et al. 2020 ) , fix seeds whenever pseudorandom numbers are involved ( Sandve et al. 2013 ) , automate for continuous integration ( Pérez-Riverol et al. 2016 ) , document dependencies ( Stojnic et al. 2020 ; Prlic & Procter 2012 ) , and to provide a README ( Stojnic et al. 2020 ; Pérez-Riverol et al. 2016 ) as well as LICENSE file ( Pérez-Riverol et al. 2016 ) . 3 An estimate of the software ecosystem state at ICML from 2014 until 2024 We have surveyed the literature and identified a set of engineering best practices, which could aid us in improving reproducibility in Machine Learning research. This section systematically looks for signs of best practice adoption within the software repositories shared in International Conference on Machine Learning papers from 2014 until 2024. Table 1: ICML conference year and the corresponding proceedings number we used for the webcrawl. International Conference on Machine Learning-year 2024 2023 2022 2021 2020 2019 2018 2017 2016 2015 2014 PMLR-volume 235 202 162 139 119 97 80 70 48 37 32 Our crawling process starts at the Proceedings of Machine Learning Research (PMLR)-website. We create links to crawl by combining the Proceedings of Machine Learning Research-processings website with the International Conference on Machine Learning volume number for each year as shown in Table 1 . Using beautiful-soup ( Richardson 2023 ) , we extract all links to papers and supplementary material, by filtering for links ending with pdf . In a next step we download the pdfs and extract all links to public GitHub pages using pdfx ( Hager 2021 ) . After downloading a beautiful-soup representation of all GitHub pages, we isolate repository pages by removing all pages without a branch-picker. Table 2: Valid crawled repository links in all accepted papers per year at the International Conference on Machine Learning International Conference on Machine Learning-year 2024 2023 2022 2021 2020 2019 2018 2017 2016 2015 2014 active-repository-links 1008 699 405 326 399 263 153 56 42 28 12 Table 2 lists the number of active repositories per conference year we can crawl in 2025. Based on the beautiful-soup representations of the repository roots, we search for files ( requirements.txt , noxfile.py , LICENSE , README.md , README.rst , tox.toml , tox.ini , setup.py , setup.cfg , pyproject.toml , environment.yml ) and folders ( test , tests , .github/workflows ) which allow us to estimate the adoption of software engineering best practices. In the interest of reproducibility, we provide the code we wrote for this analysis at https://github.com/BonnBytes/position_we_need_more_tests_in_ml . When looking at these numbers it is important to keep in mind that we are browsing the repositories in their modern 2025 state. We have no information on the state of the repositories at the time of each conference. We should, therefore, interpret these numbers with caution. Our methodology works, however, for one-time code releases, which are very common in the Machine Learning-community. 3.1 Licensing and readme files over time. Figure 2: International Conference on Machine Learning repository link crawl results for LICENSE and README files. The crawl includes accepted International Conference on Machine Learning-papers from 2014 (14) until 2024 (24). To save legend space, we use two-digit numbers for years. We add the counts for README.md and README.rst files, and show both as README . Figure 2 shows the adoption of license and readme files in software repositories over time. Fortunately, the trend in terms of readme files is up! In 2024, 95% of repositories had a readme. Unfortunately, the share of license files stagnates at around 50% over time, perpetuating the legal problems described in ( opensourceguide 2025 ) . 3.2 The adoption of Python at ICML over time. Figure 3: ICML repository link crawl results for Python adoption. We count each repository where python is listed as a langauge in GitHub’s language-box. Figure 3 shows the adoption of Python over time. The trend is pointing upwards, in 2014 only 58.3% of repositories used Python, its share has climbed to 90.4% in 2024. We believe this development bolsters the case for more rigorous software engineering. Since fewer language barriers exist, packaging especially, would help us to collaborate more effectively as a community. 3.3 Requirements documentation Figure 4: Estimated dependecy documentation over time. Is is also possible to share dependencies via setup.py or pyproject.toml files. Therefore this plot should be considered jointly with Figure 5 , which looks at packaging. Figure 4 shows the share of repositories with requirements.txt and environment.yml at International Conference on Machine Learning over time. The numbers leave much to be desired. In 2024, for example, the overall share of Python repositories was 90.4%. Unfortunately, the combined share of projects with requirements.txt and environment.yml files was only 39.2%. This number implies that the results of many projects will not be reproducible straightforwardly. We see a small boom in 2020, when the Conference on Neural Information Processing Systems code guide appeared ( Stojnic et al. 2020 ) , which asks authors to provide these files, more files start to appear. However, far too many repositories are missing requirements documentation. Especially because it is possible to provide both files automatically with a single command each. Pip users can run ⬇ pip freeze > requirements . txt to create the file. Similarly, for conda users providing the file requires typing ⬇ conda env export > environment . yml into the terminal. Afterwards both groups can simply commit the files to their code repositories. Providing these files is easy! We can do better! 3.4 Packaging adoption Figure 5: Estimated Python packaging adoption over time. Packaging as described in section 2.7 is an elegant way to improve code reusability. This section discusses our estimates of Python code packaging at International Conference on Machine Learning over time. Figure 5 illustrates that the numbers stagnate at around 20% for setup.py -files and 5% for pyproject.toml -files. The ratio is somewhat surprising, given that Python Enhancement Proposal-518 recommends the use of pyproject.toml files over setup.py files since 2016 ( Brett Cannon 2016 ) . The Python Enhancement Proposal also outlines the case against setup.py . In any event, packaging with setup.py is still better than doing nothing at all, which is what most of the community is doing. Overall, summing up the numbers for setup.py and pyproject.toml , roughly a quarter of code releases are packaged. These numbers suggest that we have room to grow! 3.5 Test and workflow use Figure 6: Systematic test adoption over time. The numbers for tox combine counts of tox.ini and tox.toml , both of which are valid filenames. Finally we turn our attention to containerized testing and workflow automation. Figure 6 illustrates our estimate of the adoption of both methods in the International Conference on Machine Learning community over time. Since containerized testing requires a correct specification of dependencies, we expect to see lower numbers. The numbers from the workflow column at the right of Figure 6 , measure the shape of repositories with a .github/workflows folder. The small numbers suggest that the community works largely without automated testing and workflow automation, even though most projects are team efforts with multiple authors. Automated testing makes teamwork easier. No one wants to be the one that breaks to build. Similarly, we don’t want to ask each other to run the tests constantly. An automated workflow allows us to delegate these tasks to a machine, which we argue is in everyone’s best interest. 4 An estimate of the Software Engineering at major 2024-Machine Learning conferences We use the OpenReview API to download all accepted papers from International Conference on Learning Representations and Conference on Neural Information Processing Systems in 2024. We then repeat the process described in section 3 . We include numbers from International Conference on Machine Learning-2024 for comparison. The numbers are based on 1008 code-repositories for International Conference on Machine Learning, 898 for International Conference on Learning Representations, and 1568 repos we found in accepted Conference on Neural Information Processing Systems papers. 4.1 Licensing and readme Figure 7: 2024 Machine Learning-conference paper repository link crawl results for LICENSE and README files. We added the counts for README.md and README.rst files. Figure 7 shows the adoption of licensing and README files over time. Surprisingly Conference on Neural Information Processing Systems has the lowest share of repositories with a license file, even though the conference organizers commendably provide the Conference on Neural Information Processing Systems code release guidelines ( Stojnic et al. 2020 ) , which ask authors to provide a license file. International Conference on Learning Representations has the best overall numbers. 4.2 The adoption of Python at major Machine Learning conferences Figure 8: 2024 Machine Learning-conference paper repository link crawl results for Python adoption. We count each repository where Python is listed as a language in Github’s language box. Figure 8 reveals that around 90% of the repository links at major conferences in 2024 contain Python code. 4.3 Requirements documentation Figure 9: 2024 Machine Learning-conference repository link shares with requirements.txt and environment.yml files. The NeurIPS call for papers asks authors to provide these files ( Stojnic et al. 2020 ) . Figure 9 counts requirements.txt and envorinment.yml files. The Conference on Neural Information Processing Systems conference organizers have explicitly communicated tips for releasing research code ( Stojnic et al. 2020 ) in their call for papers, which includes a call for authors to include both files. Unfortunately, these exemplary efforts do not lead to a higher share of these files than the file counters in code repositories at International Conference on Machine Learning or International Conference on Learning Representations. Once more, the International Conference on Learning Representations appears slightly ahead of its peer venues. 4.4 Packaging adoption Figure 10: Python-Packaging indicators at major 2024 Machine Learning conferences. We compare the ratio of Python files related to packaging in Figure 10 . Again, we see the best numbers at International Conference on Learning Representations. However, the numbers indicate that 80% of all repository authors still choose not to package their code. 4.5 Test and workflow use Figure 11: Systematic test adoption over at major 2024 Machine Learning conferences. The overall picture repeats itself once more when we consider test-folder counts, automated test-tool adoptions and workflows in Figure 11 . We see slightly more adoption at International Conference on Learning Representations, but the numbers are low overall. 5 How can we do better? When submitting or reviewing experimental work with code, we could ask ourselves a couple of key questions. A similar procedure has previously been proposed for chemoinformatics ( Hoyt et al. 2023 ) . • Does the code have a license file? • Is a readme included at the repository root? • Does the project facilitate automatic installation by others via packaging? It’s never too late to do this! When deadline pressure prevents us from spending the time early, this can still happen between paper acceptance and conference presentation. • Is it possible to run tests automatically? Did we test our external code dependencies by running a containerized set of tests? • Are all pseudorandom number generator seed values fixed? • Did automatic code checkers like flake8 or MyPy (if we are using type annotations) report any problems? Sometimes, there are good reasons for a no, which can be an acceptable answer. A mindless test crusade risks triggering authors to withhold code altogether. To improve the current situation, we should ask code authors these or similar questions more frequently when reviewing and read the author’s answers with an open mindset. If we notice that engineering best practices are not observed, we should share relevant sources from the Python community with the authors. 6 Alternative views: In artificial neural network development, systematic software engineering adds bureaucracy and slows us down Mindlessly enforcing software engineering practices when they don’t help authors would be harmful. A foolish consistency is the hobgoblin of little minds ( van Rossum et al. 2001 ; Emerson 1841 ) , we should limit ourselves to cases where systematic tests are efficient and valuable. We need the ability to accept brilliant new mathematical ideas to conferences, even if there is no accompanying code or the code is imperfect. Good theory does not require code to be valid. Furthermore, automatic tests become expensive very quickly when we evaluate artificial neural networks as part of the tests. The situation is even trickier for neural network training code, which is costly and often takes long to run. Testing neural networks is difficult. We often lack clearly defined outputs. At the same time, many Machine Learning-codes have non-deterministic properties, which makes them hard to test. Furthermore, automatic tests often become too large for online workflows, and users get feedback too late. After all, speed is crucial since groups compete with each other. Forgoing testing, dependency documentation, and packaging initially saves time, which produces a competitive advantage since skipping systematic testing will allow projects to be finished quicker and papers to appear earlier. 7 Conclusion This section concludes our call to action for more rigorous software engineering in machine learning. In the previous section, we argued that forgoing testing and dependency documentation,for example, saves time. This is probably the case for small, short-term projects. After all, there must be a reason why we did not find more test indicators in Machine Learning-project repository links. We want to use this opportunity to make the case for more rigorous research software engineering. For the following two reasons, we firmly believe that software engineering in research saves time and reaps benefits in the long run. The first reason is that scientists typically continue to work in a similar direction for years. Research is split into smaller sub-projects, which are published along the way. The sub-projects build on top of each other. Building a well-tested and packed firm foundation saves time because it reduces the need to work around bugs. Systematic testing and packaging act as an early warning and control system. The former allows us to identify problems early on and later keeps code duplication in check. This ensures that a bug, once fixed, is gone for good. Bugs in the evaluation code, for example, can invalidate entire projects. Moreover, early testing of our code protects us from embarrassing retractions later. As the network runtimes become prohibitively long, we focus our automated testing on the data pre-processing pipeline. After all, in typical conference paper projects, the gradient computation code often comes from large, carefully tested external frameworks. Consequently, we are more likely to find bugs in the preprocessing than in the network. Doing so will save us time because it is improbable to successfully train an artificial neural network on top of a buggy foundation. Our second reason is the argument that science is a team effort. As team members come and go, new team members must reproduce the numbers from the previous generation. If the last generation chose to skip dependency documentation and testing, it has accumulated technical debt, which comes due at this point. With PyTorch, for example, “reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms” ( PyTorch-Contributors 2024 ) . To reproduce results, the environment in which PyTorch ran needs to be documented. If that does not happen, our only option is to start guessing version numbers or redesigning prior work. Both activities are an avoidable waste of time and effort. Theoretical mathematical papers which rest their claim to validity on theoretical proofs are unlikely to come with repository links. It is unlikely that we see links from that group of papers in our data. Therefore, we believe the low numbers for software engineering best practice indicators can and should improve. We, for example, reuse our software all the time. This is why we believe good code quality is in our best interest. This is probably the case for many in the community. Let’s make our own lives better by improving our code. Impact Statement This paper studies the state of adoption of software engineering best practices at major Machine Learning conferences. We believe our work positively impacts the community by alerting them to methods that allow everyone to grow as a developer. We strongly believe this improves scientific research by prioritizing quality. Appendix A Appendix Acronyms Figure 12: ICML paper-link crawl results for converence papers from ICML-2014 up until ICML-2024 in a single figure.