OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents
Chenyu Zhou Xinyun Lu Jiangyue Zhao Jianghao Lin Dongdong Ge Yinyu Ye, [2pt] Shanghai Jiao Tong University Shanghai University of Finance and Economics Stanford University · 2026
Read OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents by Chenyu Zhou Xinyun Lu Jiangyue Zhao Jianghao Lin Dongdong Ge Yinyu Ye, [2pt] Shanghai Jiao Tong University Shanghai University of Finance and Economics Stanford University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28158.txt.
Introduction
Zhou and colleagues benchmark optimization agents on full industrial workflows, not one-shot problem translation. OR-Space gives agents persistent multi-file workspaces with documents, data, code, and solver outputs, across build, revise, and explain tasks. The design tests whether agents sustain reliable optimization work through a task's whole lifecycle, exposing failure modes one-shot tests hide.
Catalog record
| CREATOR | Chenyu Zhou Xinyun Lu Jiangyue Zhao Jianghao Lin Dongdong Ge Yinyu Ye, [2pt] Shanghai Jiao Tong University Shanghai University of Finance and Economics Stanford University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88855 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28158
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents Chenyu Zhou Xinyun Lu Jiangyue Zhao Jianghao Lin Dongdong Ge Yinyu Ye, [2pt] Shanghai Jiao Tong University Shanghai University of Finance and Economics Stanford University Abstract Large language model (LLM) agents are increasingly used to assist with operations research (OR) modeling, yet existing OR-oriented benchmarks often reduce evaluation to one-shot translation from a self-contained textual problem statement into a mathematical formulation or solver program. Such settings abstract away two defining characteristics of real-world industrial OR workflows: (1) the workspace setting , in which agents must operate within persistent multi-artifact workspaces containing requirements, structured data, code artifacts, solver feedback, and stakeholder interactions; and (2) the task setting , in which agents must support multiple stages of the OR lifecycle rather than only one-shot problem-to-solution generation. To this end, we introduce OR-Space , a full-lifecycle workspace benchmark for evaluating industrial optimization agents across model construction, model revision, and grounded explanation. Each OR-Space instance is represented as a persistent executable workspace containing business documents, structured data, optional code artifacts, solver outputs, and task-specific evaluators distributed across multiple interdependent files. OR-Space defines three lifecycle-oriented task modes: Build , where agents construct solver-ready optimization models from heterogeneous artifacts; Revise , where agents modify existing models under changing requirements or solver feedback while preserving valid prior logic; and Explain , where agents answer grounded questions about solutions, constraints, and business implications using evidence distributed across workspace artifacts. By combining persistent multi-artifact workspaces with lifecycle-oriented task modes, OR-Space evaluates whether agents can perform reliable optimization work beyond end-to-end text generation. We describe the benchmark design, evaluation protocol, and quality-control pipeline, and position OR-Space as a benchmark for studying the reliability, failure modes, and practical readiness of LLM agents in industrial OR workflows. 1. Introduction Operations research (OR) provides a natural testbed for evaluating the reliability of LLM agents in real-world industrial workflows. In practice, OR work rarely consists of solving a fully specified textbook-style problem in isolation. Instead, OR engineers must interpret business requirements, recover modeling assumptions from documents, extract parameters from structured data files, implement and debug solver programs, revise existing models under evolving requirements, and communicate decisions to non-expert stakeholders. These activities require not only mathematical modeling ability, but also the ability to operate reliably within persistent multi-artifact workspaces. Existing OR-oriented benchmarks have made substantial progress in evaluating whether LLMs can formulate and solve optimization problems from textual descriptions, covering linear programming, integer programming, mixed-integer programming, and nonlinear programming settings [ 13 , 19 ] . However, existing OR-oriented LLM evaluations often abstract away key characteristics of real-world industrial OR workflows along two dimensions. (1) In terms of the workspace setting , they often compress the relevant business context, data, objective, and constraints into a self-contained textual prompt. This setting is useful for testing end-to-end modeling competence, but it removes the need to locate, connect, and ground information across documents, data files, code artifacts, and solver outputs. (2) In terms of the task setting , existing evaluations often focus on static, single-shot tasks such as generating a formulation or producing solver code from a given problem statement. In practice, OR work is lifecycle-oriented: an engineer builds an initial model, revises it as business rules or data interfaces change, and explains the model and its solution to stakeholders. Agent benchmarks have demonstrated the importance of evaluating LLM agents in richer environments involving tools, files, execution state, web interfaces, and workplace-style tasks [ 16 , 46 , 23 , 35 , 5 , 32 , 36 ] . Yet these benchmarks are not designed to evaluate the domain-specific structure of industrial optimization, where correctness depends on mathematical variables, constraints, objectives, data-to-parameter mappings, solver behavior, and faithful explanations grounded in both code and business logic. As a result, current evaluations leave open a central question: can an LLM agent perform reliable OR engineering work across a multi-artifact workspace and across multiple stages of the optimization modeling lifecycle? We introduce OR-Space , a workspace-grounded benchmark for evaluating LLM agents on industrial optimization tasks. OR-Space is designed around two axes. The first is a multi-artifact workspace setting : each benchmark instance is represented as an executable workspace containing natural language documents, structured data files, optional code artifacts, solver outputs, and automated evaluation interfaces. Rather than receiving a fully serialized problem statement, an agent must recover the relevant optimization problem from distributed artifacts. The second is a lifecycle-oriented task setting : OR-Space evaluates agents across three complementary task modes, Build , Revise , and Explain . In Build , an agent constructs a solver-ready optimization model from business documents and data files. In Revise , the agent modifies an existing workspace in response to new requirements, changed assumptions, or solver feedback while preserving the valid parts of the original model. In Explain , the agent answers grounded questions about the solution, constraints, assumptions, and business implications by reasoning across documents, data, code, and solver output. This design exposes challenges that are largely hidden in exercise-style OR benchmarks. Agents must ground heterogeneous workspace artifacts into formal optimization structure, maintain cross-artifact consistency under iterative revisions, and produce explanations faithful to both the implemented model and the underlying workspace evidence. These challenges are distinct from, although complementary to, classical formulation and solving ability: an agent may successfully generate optimization code from a clean, fully specified prompt while still failing to operate reliably in realistic industrial OR workspaces. The goal of OR-Space is to support systematic evaluation of agent reliability in real-world industrial optimization workflows: identifying which stages of the OR lifecycle remain challenging, which failure modes dominate in workspace-grounded environments, and how performance changes when agents must interact with persistent multi-artifact workspaces and execution environments instead of fully specified prompts. OR-Space provides reproducible evaluation across executable model construction, targeted model revision, and grounded explanation. It further enables fine-grained attribution of failures, including execution errors, infeasible formulations, incorrect objectives, missing constraints, incomplete revisions, and unfaithful explanations. Our contributions are as follows: • We identify two limitations of existing OR-oriented LLM evaluations: self-contained workspace settings and single-shot task settings that underrepresent the multi-artifact and lifecycle-oriented nature of real-world industrial OR workflows. • We introduce OR-Space , a benchmark for industrial optimization agents built around persistent executable workspaces, where requirements, data, code artifacts, solver outputs, and evaluation interfaces are distributed across multiple files and artifacts. • We define three lifecycle-oriented task modes— Build , Revise , and Explain —capturing model construction, model maintenance, and model communication in practical industrial OR workflows. • We provide an evaluation framework for analyzing agent capabilities and failure modes in workspace-grounded OR tasks, including failures in formulation, data grounding, revision consistency, execution, and explanation faithfulness. 2. Related Work LLMs for optimization modeling. Recent OR benchmarks evaluate LLMs with solver-grounded objectives rather than textual similarity, including NL4Opt [ 29 ] , Mamo [ 15 ] , ORLM/IndustryOR [ 13 ] , OptiMUS [ 1 ] , and Chain-of-Experts/ComplexOR [ 34 ] . Newer work expands this direction through scalable data synthesis, industrial-scale model–data separation, OR question answering, specialized OR training, and agentic modeling or policy-evolution systems [ 25 , 19 , 27 , 40 , 12 , 14 ] . Taken together, these works