MADE: Benchmark Environments for Closed-Loop Materials Discovery

Shreshth A. Malik, Tiarnan Doherty, Panagiotis Tigas, Muhammed Razzak, Stephen J. Roberts, Aron Walsh, Yarin Gal · 2026

Read MADE: Benchmark Environments for Closed-Loop Materials Discovery by Shreshth A. Malik, Tiarnan Doherty, Panagiotis Tigas, Muhammed Razzak, Stephen J. Roberts, Aron Walsh, Yarin Gal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20996.txt.

Introduction

Shreshth Malik and six co-authors build proving grounds for machines that discover materials. Their 2026 paper introduces MADE, short for Materials Discovery Environments. Current benchmarks test static prediction or lone sub-tasks. Real discovery is a loop: try, learn, and try again. Static tests miss that loop in full. MADE benchmarks end-to-end autonomous discovery pipelines.

In these settings, an agent must close the loop. It proposes candidates, runs tests, and updates its plan. Scores then reflect discovery skill, not just prediction skill. The frame lets labs compare full pipelines fairly. That matters as autonomous labs move from demos to daily use. MADE gives the field a shared track for that race.

Catalog record

CREATORShreshth A. Malik, Tiarnan Doherty, Panagiotis Tigas, Muhammed Razzak, Stephen J. Roberts, Aron Walsh, Yarin Gal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20996

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MADE: Benchmark Environments for Closed-Loop Materials Discovery Shreshth A. Malik, Tiarnan Doherty, Panagiotis Tigas, Muhammed Razzak, Stephen J. Roberts, Aron Walsh, Yarin Gal Abstract Existing benchmarks for computational materials discovery primarily evaluate static predictive tasks or isolated computational sub-tasks. While valuable, these evaluations neglect the inherently iterative and adaptive nature of scientific discovery. We introduce MA terials D iscovery E nvironments ( MADE ), a novel framework for benchmarking end-to-end autonomous materials discovery pipelines. MADE simulates closed-loop discovery campaigns in which an agent or algorithm proposes, evaluates, and refines candidate materials under a constrained oracle budget, capturing the sequential and resource-limited nature of real discovery workflows. We formalize discovery as a search for thermodynamically stable compounds relative to a given convex hull, and evaluate efficacy and efficiency via comparison to baseline algorithms. The framework is flexible; users can compose discovery agents from interchangeable components such as generative models, filters, and planners, enabling the study of arbitrary workflows ranging from fixed pipelines to fully agentic systems with tool use and adaptive decision making. We demonstrate this by conducting systematic experiments across a family of systems, enabling ablation of components in discovery pipelines, and comparison of how methods scale with system complexity. 1 Introduction Figure 1 : Example acquisition curves for different discovery policies on two quinary inter-metallic systems. The acceleration factor (AF) and enhancement factor (EF) are shown for the LLM orchestrator policy with respect to a random generator baseline policy. Shaded regions are standard error in the mean across 5 episodes. Figure 2 : Conceptual overview of the MADE benchmark. a) Existing discovery pipelines and benchmarks follow a static filtering process, moving sequentially from generative models to increasingly expensive evaluation methods, without end-to-end feedback. b) MADE simulates a closed-loop discovery environment where agents iteratively propose candidates, receive oracle feedback (formation energy), and update their strategy. c) Modular, extensible components of the benchmark environments. Scientific discovery inherently runs in a closed loop. Researchers propose hypotheses, run experiments or simulations, and refine their ideas based on the outcomes ( 41 ) . Failures can be as informative as successes, and strategies adapt as new evidence emerges. Materials discovery is no exception; promising candidates are proposed, evaluated, and iteratively refined, often through cycles of exploration, dead ends, and serendipitous insights. Yet most computational benchmarks for materials discovery assume a one-way process (Figure 2 a). Predictive benchmarks evaluate accuracy of predicting properties such as band gap, energy, and forces using fixed datasets ( 15 ; 44 ; 46 ; 24 ) . Generative model evaluations on the other hand typically measure metrics such as stability, uniqueness and novelty for one-shot batch generation of candidates ( 59 ; 30 ; 8 ; 55 ) . These are valuable assessments of sub-components of pipelines, yet they evaluate models in isolation, divorced from the overall discovery workflow. In practice, researchers combine predictive models, generative models, filters, and heuristics in multi-stage pipelines, with the ultimate objective being the efficient experimental discovery of new materials. In realistic discovery settings, evaluations such as a high-fidelity simulation or physical experiment can be very costly. Thus the efficiency of discovery algorithms under a limited query budget is important. Methods such as Bayesian optimization and active learning ( 47 ; 16 ) provide principled frameworks for adaptive experimentation and have shown strong performance in molecular and materials applications ( 28 ; 45 ) . These approaches differ in their modeling and acquisition strategies, but are typically developed in settings focused on efficiently optimizing a small number of continuous design variables toward a single global objective. Materials discovery, by contrast, is a multi-minima seeking problem that aims to find diverse stable or metastable compounds within a vast, discrete and constrained chemical space. Concurrently, LLM-based scientific agents have demonstrated increasing capability at orchestrating multi-step workflows, integrating prior knowledge, and adapting strategies given feedback ( 29 ; 21 ; 17 ; 36 ; 1 ) . While some recent benchmarks begin to probe aspects of scientific reasoning and tool use ( 51 ; 31 ) , there remains limited consensus on how to systematically evaluate agentic systems in open-ended, feedback-driven discovery settings ( 49 ) . Materials discovery provides a natural testbed for such systems: candidate proposals can be computationally verified in a closed loop, and there exists a rich ecosystem of computational tools and prior literature. This can enable controlled evaluation of discovery-relevant agentic behaviors such as planning and adaptive decision making. To address these gaps in evaluations, we introduce a family of MA terials D iscovery E nvironments ( MADE ) that enable the evaluation of closed-loop discovery pipelines. In MADE, an agent or algorithm sequentially proposes candidate structures, receives feedback from the environment, and adjusts its strategy to efficiently discover novel thermodynamically stable compounds under a limited query budget. MADE is intentionally modular and composable; users can combine arbitrary planners, generators, filters, and scorers into pipelines, or evaluate fully agentic systems that can utilize environmental feedback. All experiments are specified via configuration files, facilitating reproducible comparisons and systematic ablations across discovery strategies. Crucially, MADE supports discovery-centric evaluation metrics ( 14 ; 2 ) . We quantify how quickly a method discovers new materials with respect to a baseline search strategy, extending the discovery-acceleration paradigm beyond machine learning interatomic potential (MLIP) screening ( 44 ) to the full discovery loop. This enables answering questions such as: what is the performance gain of using a better generative model? Do surrogate model rankings meaningfully accelerate discovery? Are agentic LLM systems more efficient than traditional search algorithms? In summary, our contributions are as follows: • We introduce MADE, a family of environments for benchmarking closed-loop computational materials discovery, enabling the first systematic evaluation of full pipelines on open-ended discovery metrics. • We benchmark strategies ranging from random search with generative models to agentic systems, across multiple system complexities, ablating the contributions of different components in pipelines. • We find that agentic systems and adaptive search algorithms become more important for discovery efficiency as chemical complexity and search spaces scale, and as surrogate models reduce in efficacy. 2 MADE: MAterials Discovery Environment We first define criteria and motivation for the design of our benchmark. We then introduce MADE 1 1 1 https://github.com/diffractivelabs/MADE , the proposed framework for evaluating materials discovery strategies. 2.1 Desiderata for Discovery Benchmarks We argue that a benchmark for materials discovery should satisfy three general desiderata: • Evaluate closed-loop discovery . It should directly measure how effectively an end-to-end closed-loop system finds new materials to enable ablation of components in the pipeline. • Reflect realistic search challenges in materials science . It should reflect the discrete, structured but sparse, multi-minima landscape of materials discovery. • Be general, scalable, and method-agnostic . It should support controlled experiments across chemical systems, search-space sizes, and fidelity levels while remaining implementation-agnostic. We use these desiderata to motivate MADE’s design while remaining independent of specific pipeline implementation choices. We compare MADE to existing discovery benchmarks against these criteria in Appendix A . 2.2 Problem Definition In MADE, an agent or algorithm interacts with a structured chemical environment by proposing candidate materials, receiving oracle feedback, and adapting its strategy over time. The goal is to uncover new thermodynamically stable compounds efficiently

Read the complete text