Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling
Yitian Chen, Jingfan Xia, Siyu Shao, Dongdong Ge, Yinyu Ye · 2025
Read Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling by Yitian Chen, Jingfan Xia, Siyu Shao, Dongdong Ge, Yinyu Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11792.txt.
Introduction
Yitian Chen and a team of co-authors present a training method that grounds language models in real solvers, in two thousand twenty-five. The task is turning plain language into a formal optimization model that a solver can run. Models often invent flawed models that look right but fail in practice.
Their method uses the solver itself as a checker during learning. It grades the model's code and its math file for syntax, feasibility, and solution quality. Those grades become direct rewards for training. The checker also helps build clean training data by a self-consistency trick. Tests on public benchmarks beat prior methods at producing accurate, runnable models.
Catalog record
| CREATOR | Yitian Chen, Jingfan Xia, Siyu Shao, Dongdong Ge, Yinyu Ye |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11792
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling Yitian Chen, Jingfan Xia, Siyu Shao, Dongdong Ge, Yinyu Ye Abstract Optimization modeling is fundamental to decision-making in fields such as supply chain management, logistics, and financial engineering, but its complexity presents a major barrier to adoption. Automating model creation from natural language is key to improving efficiency and access. However, while Large Language Models (LLMs) are a promising tool for this, they often produce flawed or infeasible results due to errors and hallucinations. To address this issue, we propose Solver-Informed Reinforcement Learning (SIRL), a framework that uses Reinforcement Learning with Verifiable Reward to improve LLMs’ ability to generate accurate and executable optimization models. Specifically, SIRL automatically assesses the executable code and the instance-level mathematical model represented by the associated .lp files. This process yields precise feedback on syntactic validity, feasibility, and solution quality, which serves as a direct reward signal to guide the reinforcement learning process. Furthermore, this verification mechanism also supports our instance-enhanced self-consistency method for creating high-quality training data. Extensive experiments on diverse public benchmarks demonstrate that models trained with our SIRL framework achieve state-of-the-art performance, substantially outperforming existing methods in generating accurate and executable optimization models. Specifically, our SIRL-32B model surpasses DeepSeek-V3 and OpenAI-o3 on the majority of these benchmarks. Our code is publicly available at https://github.com/Cardinal-Operations/SIRL . 1 Introduction Optimization modeling provides a powerful framework for decision-making across diverse fields, from logistics and finance to engineering and machine learning Singh (2012) ; Lu (2007) ; Mokhtar et al. (2014) ; Cornuejols and Tütüncü (2005) ; Bishop and Nasrabadi (2006) . The process involves two primary steps: first, converting a natural language problem description into a formal mathematical model, and second, employing a solver to solve an optimization problem to get the optimal solution. Despite the maturity and powerful capabilities of modern optimization solvers, such as Gurobi Gurobi Optimization, LLC (2024) , COPT Ge et al. (2022) and CPLEX Ilog (2010) , formulating these intricate real-world problems into precise mathematical models and executable codes remains a significant bottleneck, often requiring substantial domain expertise and considerable manual effort Lu (2007) ; Luenberger et al. (1984) . To bridge this gap, modeling languages such as MiniZinc Nethercote et al. (2007) and Essence Frisch et al. (2008) emerged, offering a high-level declarative syntax to abstract away low-level solver details, particularly for constrained programming. However, even with these languages, the crucial first step, transforming a natural language problem description into a correct and efficient model, remains a formidable manual task Tsouros et al. (2023) . The advent of Large Language Models (LLMs), (e.g., GPTs Achiam et al. (2023) , Gemini Team et al. (2023) ; Team et al. (2024) , Deepseek Liu et al. (2024) ), offers a promising avenue to automate or assist in this mathematical modeling and code generation process, potentially democratizing access to optimization solvers. However, ensuring the correctness, feasibility, and solver-compatibility of LLM-generated optimization models presents a significant and increasingly active research challenge. In general, existing approaches that leverage LLMs for optimization modeling can fall broadly into two categories. Firstly, prompt-based or agent-based approaches utilize the frozen capabilities of powerful foundation LLMs Achiam et al. (2023) ; Team et al. (2024) ; Team et al. (2023) ; Liu et al. (2024) , guide the models to extract relevant problem information, and generate the corresponding mathematical models and executable codes for optimization solvers Ramamonjison et al. (2022) ; Xiao et al. (2023) ; AhmadiTeshnizi et al. (2024) ; Astorga et al. (2024) . Although accessible, these methods do not adapt the underlying model parameters and can be sensitive to prompt design and the chosen foundation LLMs. A distinct line of work addresses training open-source LLMs Grattafiori et al. (2024) ; Yang et al. (2024a) to enhance their capabilities for optimization modeling using offline learning approaches. This encompasses techniques such as Supervised Fine-Tuning (SFT) on demonstrations data, and alignment techniques (e.g., DPO Rafailov et al. (2024) , KTO Ethayarajh et al. (2024) ) on preference data. The success of existing offline methods Huang et al. (2025) ; Lu et al. (2025) ; Jiang et al. (2024) hinges on the availability of carefully curated datasets. These datasets, comprising problem descriptions paired with either detailed mathematical models and code demonstrations or comparative human preference labels, are typically generated through human annotation Jiang et al. (2024) or synthesis Huang et al. (2025) ; Lu et al. (2025) . Training on these curated datasets enables offline learning approaches to capture the data’s stylistic and structural patterns, such as mathematical formulation and solver code, and achieve good performance. However, since their training objective focuses on mimicking demonstrations or aligning preferences, these methods still struggle to inherently guarantee functional correctness or solution feasibility, which are essential for reliable solver execution. Recently, the development of powerful Large Reasoning Models (LRMs) marks a major leap in the reasoning capabilities of LLMs OpenAI () ; Guo et al. (2025) ; Team et al. (2025) . The process begins with Chain-of-Thoughts (CoTs) prompting Wei et al. (2022) , which unlocked the ability for LLMs to generate explicit, multi-step reasoning paths. Building on this, Reinforcement Learning with Verifiable Rewards (RLVR) Guo et al. (2025) ; Lambert et al. (2024) provides the essential feedback mechanism to ensure the correctness of these paths. RLVR directly optimize LLMs’ policies using objective feedback from verified outcomes, such as the validation of mathematical solutions or the execution of generated code against test cases Lambert et al. (2024) ; Guo et al. (2025) ; Team et al. (2025) ; Li et al. (2025b) ; Feng et al. (2025) . This powerful combination has led to LRMs OpenAI () ; Guo et al. (2025) ; Team et al. (2025) that excel at complex reasoning, demonstrated by high performance on challenging mathematical competitions like the AIME and the International Mathematical Olympiad (IMO) Guo et al. (2025) ; OpenAI () . For optimization tasks, solving a real-world problem involves a multistep process encompassing problem analysis and reasoning, mathematical modeling, and code implementation Ramamonjison et al. (2022) ; Xiao et al. (2023) ; AhmadiTeshnizi et al. (2024) ; Astorga et al. (2024) ; Huang et al. (2025) ; Lu et al. (2025) ; Jiang et al. (2024) , resembling a CoT reasoning process. The outputs of this process–the mathematical model and the solver code–can be verifiable using external optimization solvers. Verification involves steps such as syntax checks, feasibility assessment through model solving, and comparison of the objective value against known optimal value. These verifiable checks yield objective and richness reward signals, enabling RLVR Shao et al. (2024) ; Guo et al. (2025) ; Lambert et al. (2024) ; Yang et al. (2023) to directly optimize the LLM generation towards producing correct, feasible, and high-quality outputs for complex optimization problems. To the best of our knowledge, this is the first application of RLVR to directly enhance LLMs’ proficiency in optimization modeling. The rewards, including feasibility status, objective function value, and mathematical model statistics from the .lp file, are obtained by executing the generated code. Specifically, the .lp file, which is a standard format for optimization solvers, provides an explicit instance-level representation of the generated mathematical model, implicitly reflecting the LLM’s