Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Network
Theresa Eimer, Lennart Schäpermeier, André BiedenkappAlexander Tornede, Lars Kotthoff, Pieter Leyman, Matthias Feurer, Katharina Eggensperger, Kaitlin MaileTanja Tornede, Anna Kozak, Ke Xue, Marcel Wever, Mitra Baratchi, Damir Pulatov, Heike Trautmann, Haniye Kashgarani, Marius Lindauer · 2025
Read Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Network by Theresa Eimer, Lennart Schäpermeier, André BiedenkappAlexander Tornede, Lars Kotthoff, Pieter Leyman, Matthias Feurer, Katharina Eggensperger, Kaitlin MaileTanja Tornede, Anna Kozak, Ke Xue, Marcel Wever, Mitra Baratchi, Damir Pulatov, Heike Trautmann, Haniye Kashgarani, Marius Lindauer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16491.txt.
Introduction
Theresa Eimer leads a large author team in codifying research practice in an arXiv report made public in 2025. Meta-algorithmic research covers algorithm selection, configuration, and scheduling. It runs huge experiments with many design choices. Each choice is a possible source of error. Those errors threaten the validity of the findings.
The report gathers best practices from across the COSEAL research network. Its subfields had been evolving their standards separately until now. The guidance walks the whole experimental cycle. It starts with framing research questions and picking a design. It goes on to running experiments, then analyzing and presenting results fairly. The result is a shared guideline for newcomers and practitioners alike. By unifying scattered advice, the authors hope to cut wasted computation. They also aim to make results in the field easier to trust and reproduce.
Catalog record
| CREATOR | Theresa Eimer, Lennart Schäpermeier, André BiedenkappAlexander Tornede, Lars Kotthoff, Pieter Leyman, Matthias Feurer, Katharina Eggensperger, Kaitlin MaileTanja Tornede, Anna Kozak, Ke Xue, Marcel Wever, Mitra Baratchi, Damir Pulatov, Heike Trautmann, Haniye Kashgarani, Marius Lindauer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-16491
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Network Theresa Eimer, Lennart Schäpermeier, André BiedenkappAlexander Tornede, Lars Kotthoff, Pieter Leyman, Matthias Feurer, Katharina Eggensperger, Kaitlin MaileTanja Tornede, Anna Kozak, Ke Xue, Marcel Wever, Mitra Baratchi, Damir Pulatov, Heike Trautmann, Haniye Kashgarani, Marius Lindauer Abstract Empirical research on meta-algorithmics, such as algorithm selection, configuration, and scheduling, often relies on extensive and thus computationally expensive experiments. With the large degree of freedom we have over our experimental setup and design comes a plethora of possible error sources that threaten the scalability and validity of our scientific insights. Best practices for meta-algorithmic research exist, but they are scattered between different publications and fields, and continue to evolve separately from each other. In this report, we collect good practices for empirical meta-algorithmic research across the subfields of the COSEAL community, encompassing the entire experimental cycle: from formulating research questions and selecting an experimental design, to executing experiments, and ultimately, analyzing and presenting results impartially. It establishes the current state-of-the-art practices within meta-algorithmic research and serves as a guideline to both new researchers and practitioners in meta-algorithmic fields. 1 Improving Experimental Design in Meta-Algorithmic Research Empirical meta-algorithmic research allows for a large degree of control over the experimental setup due to the computational nature of our experiments. However, this is a double-edged sword for research that is both meaningful and efficient: We encounter a plethora of decisions that must be made deliberately every step of the way. This can lead to divergent research practices and results, which in turn can diminish the amount of meaningful progress made in a field, as common standards for what constitutes reliable research results are important for long-term progress - compare 4 ; 102 for the result of too much variation in a research field with too little systematic comparison. In addition, we increasingly need to consider the impact our research has on other disciplines and society at large. Practices to ensure fairness and accountability in application ( 38 ; 61 ; 117 ) , are not yet well integrated into meta-algorithmic research. Therefore, conducting meta-algorithmic research with reliable and reproducible outcomes for the different dimensions relevant for real-world applications is an ever-increasing challenge. Our Goals are to provide a written account of ‘‘common knowledge’’ principles and best practices from the COSEAL 1 1 1 COSEAL ( CO nfiguration and SE lection of AL gorithms) is an international research community on meta-algorithmic topics with more than a decade of experience on empirical research; see also https://www.coseal.net/ . community to improve the experimental design in meta-algorithmic research in order to promote more research that matters long-term for real-life applications. Specifically, we focus on: 1. Drawing empirically sound conclusions through carefully designed evaluation pipelines. 2. Improving efficiency of research workflows to enable more and better scientific insights throughout the field. 3. Increasing trust in research results through applicability, robustness, fairness, and accountability. The Structure of this paper mimics the typical research process in meta-algorithmics. We begin by discussing the process of designing research questions . We move on to construct an empirical evaluation for these research questions through high-level experimental design . Next, we investigate software , as it is our primary tool: helpful code practices, working with benchmarks and dependencies, and other important considerations for research software. Lastly, we discuss interpreting results , including visualization of results and statistical testing. All sections include examples of good research practices, as well as common pitfalls to avoid. We believe this framework is a useful guide to researchers at any level of experience and can serve as a structured introduction to conducting meta-algorithmic research. Nevertheless, it is not our goal to provide a rule book or a collection of best practices, since specific practices in each field evolve over time. Indeed, the question of how to do computational research is a core part of our research field itself. As such, we encourage readers to challenge our recommendations and add their findings to this living document 2 2 2 Contributions & Questions are managed via GitHub: https://github.com/coseal/COSEAL-Best-Practices . Similarly, we commend thoughtful breaking with the practices we present if they do not serve the purpose of the research at hand, as in the spirit of furthering scientific practice, researchers occasionally “should forget the textbook and simply ask whether he has achieved these higher purposes” ( 70 ) . 2 Formulating Research Questions Before starting empirical work or experiment design, it is crucial to clarify the objectives of a research project. This section makes general suggestions for doing this, specifically by discussing best research practices and formulating research goals. TL;DR: Be familiar with the literature on reproducibility and best experimental practices in your field. Decide at the start whether your research project will be confirmatory or exploratory. Start with a clear research question or hypothesis. 2.1 Read Best Practice Literature The first step of any research project should be to know the best practices in the field. Without this background knowledge, even finding interesting research questions can be challenging, let alone designing an experimental evaluation. Building on a field’s established norms, on the other hand, can not only improve a project’s outcomes, but also save a lot of research and review effort. Much research has studied best practices for empirical evaluations and experimental design in and around meta-algorithmics (see, e.g. 89 ; 49 ; 56 ; 84 ; 35 ; 76 ) . Such best practices can contain a range of recommendations to be aware of: from high-level discussions of research directions (see the conversation around metaphor-based meta-heuristics by 4 and supporting benchmarking results by 123 ) to general points about the validity of theoretical and empirical results ( 49 ; 56 ; 47 ) and practical questions of reproducibility and replicability ( 88 ; 97 ; 16 ) , see Table 1 . Many domains have their own set of best practices in addition to general recommendations for meta-algorithmic research - some like, e.g., algorithm configuration ( 35 ) , with highly developed standards we can learn from. Therefore, we recommend reading up on how a specific community has formulated its best practices and where they might differ from others. This should form a strong basis for formulating interesting research questions that move the field forward and validating them empirically. We want to add a note on AI assistance at this point since it is relevant here but also for the rest of this paper. Many researchers from different disciplines use AI tools to help with different research tasks, like summarizing literature, improving writing or creating figures. It is certainly worth exploring if AI tools can improve your personal workflows, but especially when it comes to literature research, you should always be able to verify, explain and reproduce the result from an AI tool. This means reading the literature first before using AI summaries, thorough corrections of AI-assisted scientific writing and detailed scrutiny of other generated research artifacts like figures. Also, note the data protection issues associated with many AI services. When using AI tools, we recommend systems hosted by trusted institutions (many universities now offer such internal services) or systems that allow for some measure of control over data usage rights.