Towards Community-Driven Agents for Machine Learning Engineering
Sijie Li, Weiwei Sun, Shanda Li, Ameet Talwalkar, Yiming Yang · 2025
Read Towards Community-Driven Agents for Machine Learning Engineering by Sijie Li, Weiwei Sun, Shanda Li, Ameet Talwalkar, Yiming Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20640.txt.
Introduction
Sijie Li and co-authors build AI agents that learn from a coder crowd in a 2025 paper. Lone agents grind on one task and miss the tips a crowd trades. Their test bed, MLE-Live, fakes a live contest crowd the agent can read and ask. On top sits CoMind, a team of agents that grows many drafts at once. Breadth and depth stay in balance that way.
On 75 past contests in the bed, CoMind took medals in 36 percent of runs, a new best mark. Sent into eight live contests, it beat 92.6 percent of human coders on mean rank. It hit the top five percent on three boards and the top one percent on one. The authors say crowd sense is the edge lone agents lack.
Catalog record
| CREATOR | Sijie Li, Weiwei Sun, Shanda Li, Ameet Talwalkar, Yiming Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-20640
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Community-Driven Agents for Machine Learning Engineering Sijie Li, Weiwei Sun, Shanda Li, Ameet Talwalkar, Yiming Yang Abstract Large language model-based machine learning (ML) agents have shown great promise in automating ML research. However, existing agents typically operate in isolation on a given research problem, without engaging with the broader research community, where human researchers often gain insights and contribute by sharing knowledge. To bridge this gap, we introduce MLE-Live, a live evaluation framework designed to assess an agent’s ability to communicate with and leverage collective knowledge from a simulated Kaggle research community. Building on this framework, we propose CoMind, a novel agent that excels at exchanging insights and developing novel solutions within a community context. CoMind achieves state-of-the-art performance on MLE-Live and outperforms 79.2% human competitors on average across four ongoing Kaggle competitions. Our code is released at https://github.com/comind-ml/CoMind . 1 Introduction Large language model (LLM)-based agents have shown remarkable potential in automating complex reasoning and decision-making tasks, ranging from software engineering ( Jimenez et al., 2023a ; Xia et al., 2025 ) and mathematical problem solving ( OpenAI, 2024 ; Ren et al., 2025 ; Li et al., 2025 ) to scientific research exploration ( Romera-Paredes et al., 2024 ; Yamada et al., 2025 ; Sun et al., 2025 ; Feng et al., 2025 ) . Among these domains, machine learning engineering (MLE) remains a particularly impactful yet challenging application area, requiring design, implementation, and evaluation of high-performing models across diverse data science tasks. Recent advances have introduced LLM agents capable of autonomously developing machine learning pipelines for Kaggle-style competitions ( Chan et al., 2025 ) . For example, MLAB ( Huang et al., 2024 ) adopts a ReAct-style ( Yao et al., 2023 ) agent for structured decision-making across tasks. AIDE ( Jiang et al., 2025 ) leverages tree-based exploration for improved efficiency, and AutoKaggle ( Li et al., 2024 ) introduces a multi-agent system with skill specialization. These agents have made progress toward end-to-end automation of MLE. However, existing systems typically operate in isolation, relying solely on internal memory and trial-and-error exploration while ignoring a critical component of real-world scientific practice — community knowledge sharing. In real data science competitions and research workflows, participants frequently learn from public discussions, shared notebooks, and community insights. Such collective knowledge significantly enhances solution quality and innovation. Current agents, due to the inability to engage with this dynamic external context, often converge to repetitive strategies and plateau in performance. Therefore, we are motivated to explore the following critical research question: How can we evaluate and design research agents that utilize collective knowledge? To address this question, we introduce MLE-Live , a novel live evaluation framework simulating a Kaggle-style research community. Unlike prior benchmarks, MLE-Live includes time-stamped public discussions and shared code artifacts available before competition deadlines – resources that human participants routinely leverage. This setup allows us to rigorously evaluate an agent’s ability to use community knowledge in a realistic, temporally grounded setting. In addition, MLE-Live supports both offline evaluation on past competitions and online evaluation on ongoing competitions, enabling comprehensive assessment across static and dynamic scenarios. Figure 1: CoMind achieves strong performance on four ongoing Kaggle competitions. Performance is measured by win rate aginst human participants. Building upon this framework, we propose CoMind , a new LLM-based MLE agent that systematically explores diverse ideas, iteratively refines solutions, and selectively incorporates external knowledge. CoMind maintains an evolving idea pool and constructs multiple distinct solution drafts in parallel. It dynamically focuses on one draft at a time, enabling efficient implementation without prompt overflow while preserving technical accuracy. Inspired by human brainstorming, this design balances exploratory breadth with practical depth. We evaluate CoMind in both previous and ongoing data science competitions. For evaluation on previous competitions, CoMind is tested on the MLE-Live benchmark, which includes 20 past Kaggle competitions across 7 domains. CoMind achieves state-of-the-art performance, significantly outperforming prior agents such as AIDE. For evaluation on ongoing competitions, we deploy CoMind on four ongoing Kaggle competitions, where it outperforms 79.2% human competitors on average (Figure 1 ), demonstrating strong real-world practicality. Human evaluations further confirm that CoMind generates more sophisticated and longer code, reflecting deeper reasoning and better integration of novel insights. In summary, our contributions are: • MLE-Live : A live evaluation framework simulating community-driven machine learning research, including shared discussions and code for realistic agent benchmarking. • CoMind : A novel LLM-based agent that excels at leveraging collective knowledge and iterative idea exploration, achieving medal-level performance in real Kaggle competitions. • Agent design innovations : We propose a new iterative and parallel exploration mechanism that enables continuous knowledge accumulation, improves diversity, and overcomes LLM context window limitations. 2 Related Work The rise of large language models (LLMs) has sparked a new wave of research into LLM-driven agents — systems that leverage LLMs’ reasoning and language capabilities to autonomously perceive, plan, and act within digital or physical environments. Early works such as ReAct ( Yao et al., 2023 ; Schick et al., 2023 ; Shen et al., 2023 ; Hong et al., 2023 ; Boiko et al., 2023 ) introduced frameworks that transform LLMs into programmable reasoning engines by interleaving natural language reasoning with tool-use actions. Subsequent studies have extended these agents to various domains, including computer usage ( Xie et al., 2024 ; Zhou et al., 2024 ) and software development ( Wang et al., 2025 ; Jimenez et al., 2023b ) . In parallel, the field of automated machine learning (AutoML) aims to reduce human involvement in building ML pipelines by automating tasks such as model selection, hyperparameter tuning, and architecture search. Early systems like Auto-WEKA ( Thornton et al., 2013 ) , HyperBand ( Li et al., 2018 ) and Auto-sklearn ( Feurer et al., 2022 ) used early stopping and Bayesian optimization to search over pipeline configurations, while methods like DARTS ( Liu et al., 2019 ) expanded automation to neural architectures. More recent frameworks such as AutoGluon ( Erickson et al., 2020 ) and FLAML ( Wang et al., 2021 ) emphasize efficiency and ease of use. Building on these developments, recent efforts have applied LLM-based agents to machine learning engineering (MLE) tasks ( Hollmann et al., 2023 ; Guo et al., 2024 ; Li et al., 2024 ; Grosnit et al., 2024 ; Hong et al., 2024 ; Chi et al., 2024 ; Trirat et al., 2024 ; Huang et al., 2024 ) . However, most evaluations remain constrained to closed-world settings with predefined search spaces, offering limited insight into how these agents perform in open-ended or collaborative ML environments. While some agents ( Guo et al., 2024 ; AI-Researcher, 2025 ) incorporate basic retrieval tools, these are typically based on simple semantic matching, and robust evaluation methodologies remain underdeveloped. Meanwhile, several benchmarks have been proposed to evaluate machine learning (ML) engineering capabilities. MLPerf ( Mattson et al., 2020 ) assesses system-level performance, including training speed and energy efficiency. To evaluate end-to-end ML workflows, MLAB ( Huang et al., 2024 ) tests the capabilities of LLM-based agents across 13 ML tasks. MLE-Bench ( Chan et al., 2025 ) and DSBench ( Jing et al., 2025 ) further extends to about 75 Kaggle competitions covering tasks