Automated Model Discovery via Multi-modal & Multi-step Pipeline
Lee Jung-Mok, Nam Hyeon-Woo, Moon Ye-Bin, Junhyun Nam, Tae-Hyun Oh · 2025
Read Automated Model Discovery via Multi-modal & Multi-step Pipeline by Lee Jung-Mok, Nam Hyeon-Woo, Moon Ye-Bin, Junhyun Nam, Tae-Hyun Oh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25946.txt.
Introduction
Lee Jung-Mok and co-authors present a pipeline for automated model discovery in a paper from two thousand twenty-five. Model discovery means searching a large space of candidate models to fit a data set well. Good models must capture fine detail yet generalize beyond the training data, without growing too complex.
Their pipeline uses two vision-language modules in an agent-like loop. One plans analyses over several steps and proposes candidate models. The other judges the candidates on numbers and on visual fit, weighing local detail against overall trend. Tests showed the pipeline finds models that balance both aims. Further checks confirmed that using images and multiple steps each play a key role.
Catalog record
| CREATOR | Lee Jung-Mok, Nam Hyeon-Woo, Moon Ye-Bin, Junhyun Nam, Tae-Hyun Oh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-25946
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automated Model Discovery via Multi-modal & Multi-step Pipeline Lee Jung-Mok, Nam Hyeon-Woo, Moon Ye-Bin, Junhyun Nam, Tae-Hyun Oh Abstract Automated model discovery is the process of automatically searching and identifying the most appropriate model for a given dataset over a large combinatorial search space. Existing approaches, however, often face challenges in balancing the capture of fine-grained details with ensuring generalizability beyond training data regimes with a reasonable model complexity. In this paper, we present a multi-modal & multi-step pipeline for effective automated model discovery. Our approach leverages two vision-language-based modules (VLM), AnalyzerVLM and EvaluatorVLM, for effective model proposal and evaluation in an agentic way. AnalyzerVLM autonomously plans and executes multi-step analyses to propose effective candidate models. EvaluatorVLM assesses the candidate models both quantitatively and perceptually, regarding the fitness for local details and the generalibility for overall trends. Our results demonstrate that our pipeline effectively discovers models that capture fine details and ensure strong generalizability. Additionally, extensive ablation studies show that both multi-modality and multi-step reasoning play crucial roles in discovering favorable models. 1 Introduction Model discovery aims to identify the optimal model structure and parameters that best represent given data. Historically, model discovery has been conducted manually by scientists [ 40 , 41 ] and has established groundbreaking advances in science and technology that have shaped our understanding of the world. However, as the complexity and scale of modern datasets continue to grow, the feasibility of manual model discovery has become increasingly limited [ 46 , 11 , 58 ] . Automating this process [ 7 , 46 , 14 , 33 , 23 , 38 ] offers the potential to accelerate the scientific progress by reducing reliance on human experts and efficient exploring of complex model spaces. Nevertheless, automatically finding the appropriate model structure is inherently challenging as it requires: 1) exploring over a vast combinatorial search space of candidate models, and 2) balancing between interpretability and model fit, ensuring the model captures the data accurately while remaining sufficiently simple and understandable to domain experts. Most existing systems [ 29 , 14 , 33 , 43 , 8 ] were carefully designed to address these challenges. For instance, [ 14 ] and [ 33 ] proposed a predefined grammar for kernel composition within Gaussian processes, which structured the search process and reduced the manual effort required to determine kernel composition. Also, the objective function relied on predefined quantifiable metrics for the model selection, rather than dynamically adapting criteria as a human expert would. However, designing the balanced and sophiscate procedure for composing models requires significant modeling expertise, which compromises the flexibility needed for automation. To achieve a more flexible and intelligent model discovery system, we substitute human experts’ roles in the model discovery with multi-modal agents. First, we introduce AnalyzerVLM, an agent that is capable of conducting in-depth analysis of given data and models through multi-step reasoning. We task AnalyzerVLM with generating code for analyzing the current model and proposing improved candidate models based on its findings. We observe that AnalyzerVLM actively uses existing libraries such as NumPy [ 42 ] and Matplotlib [ 21 ] , which are commonly used by humans for analysis and visual inspection of plots. AnalyzerVLM iteratively analyzes the current model and data until it identifies candidate models that can outperform the given model. Second, we propose a Visual Information Criterion (VIC) performed by EvaluatorVLM. Our new criterion is designed to incorporate the way humans perceive a given model. Traditional criteria, e.g ., Bayesian Information Criterion (BIC), quantify both goodness-of-fits and model complexity to identify undesirable models, but we found that they are often counter-intuitive and fail to consider inherent data characteristics and relationships. On the other hand, humans easily identify inherent data and model trends, structures, and patterns through visualization. By introducing visual representation into the model selection criteria, our VIC complements traditional criteria, enabling the discovered model to escape suboptimal. By alternating the AnalyzerVLM and EvaluatorVLM, we demonstrate that our multi-step and multi-modal pipeline effectively searches over the model space like a human, enhancing the proposed model’s quality and showing high generalizability compared to other model discovery methods. 2 Related Work Figure 1: Overall pipeline. Our multi-modal & multi-step pipeline contains 4 stages: model proposal, model fitting, model evaluation, and model selection. During the model proposal, AnalyzerVLM repeatedly analyzes the data until it determines the results are sufficient. Then, it proposes a model structure with linearity(LIN) and periodicity(PER), then the model fitting is conducted. In the model evaluation stage, EvaluatorVLM assesses models visually and utilizes the score in the subsequent model selection process. Automated Model Discovery Model discovery typically includes tasks such as selecting the model structure, estimating parameters, validating the model’s effectiveness, and refining the model. The discovery process requires extensive expertise across multiple data domains and involves multiple steps. Thus, model discoveries require human experts to repeatedly refine the model and reflect the data domain’s prior knowledge. To alleviate these difficulties, automated model discovery processes have been proposed. The goal is to reduce the burden of needing human experts and prior work has been developed by defining model space on Gaussian process [ 14 , 33 , 22 , 26 , 4 , 27 , 6 , 15 , 35 ] . Such methods are based on kernel structure search with pre-defined grammar, iteratively evolving its kernel structure. Automatic model discovery pipeline often uses the greedy-search based approaches [ 14 , 33 ] , sampling-based approaches [ 44 , 45 ] , and employing large language models (LLMs) [ 30 ] . Also there are similar approaches in symbolic regression [ 2 , 19 , 60 , 69 , 59 , 5 , 25 , 28 , 31 ] , which searches for the function composition that most fits to the data. Given a basis function and the operators, it automatically finds out the appropriate function composition over evolving. Symbolic regression requires diverse function candidates to be generated and takes the evolving over the generation, it mostly utilizes genetic programming [ 2 , 19 , 60 , 69 ] , or neural networks such as transformers [ 5 , 24 , 55 , 31 ] . These days symbolic regression utilizing LLMs [ 16 , 51 , 48 , 39 ] . Thus, utilizing LLMs is promising in the automatic model discovery pipeline, as LLMs can be adapted to those with broad expert knowledge by providing a suitable prompt. In this work, we extend the usage of LLMs to VLMs by effectively harnessing the reasoning, planning, and evaluating capabilities with visual modality. LLMs & VLMs-based Data Understanding Interpreting and understanding current data is crucial for model discovery and prediction, as extrapolation regimes including future events often depend on past patterns. [ 17 ] show that LLM has the ability to effectively deal with time-series data if the numeric tokens are properly designed. [ 13 ] propose to use plot visualizations instead of lengthy texts. [ 30 ] simply investigate the potential of visual plots as a replacement of text representations to reduce the number of tokens, and shows the comparable alternative to the text inputs. Like such methods for understanding data, there are also automatic data-driven discovery frameworks [ 37