Evaluating LLM Reasoning in the Operations Research Domain with ORQA
Mahdi Mostajabdaveh, Timothy T. Yu, Samarendra Chandan Bindu Dash, Rindranirina Ramamonjison, Jabo Serge Byusa, Giuseppe Carenini, Zirui Zhou, Yong Zhang · 2024
Read Evaluating LLM Reasoning in the Operations Research Domain with ORQA by Mahdi Mostajabdaveh, Timothy T. Yu, Samarendra Chandan Bindu Dash, Rindranirina Ramamonjison, Jabo Serge Byusa, Giuseppe Carenini, Zirui Zhou, Yong Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17874.txt.
Introduction
Operations research experts turn word problems into math models, a skill that tests whether language models truly reason in technical domains. Mahdi Mostajabdaveh and seven co-authors built a benchmark for that skill in a 2024 paper, called ORQA. The problems come from real optimization work and were written by specialists in the field.
Each problem demands several reasoning steps before the model can even be stated, let alone solved. The team ran a range of open language models against the set, including LLaMA, DeepSeek, and Mixtral builds. Scores were modest across the board. The gap between fluent answers and expert-grade modeling stood out as the main finding. The dataset and scoring code are public, and the authors offer the benchmark as a yardstick for progress in machine reasoning about optimization.
Catalog record
| CREATOR | Mahdi Mostajabdaveh, Timothy T. Yu, Samarendra Chandan Bindu Dash, Rindranirina Ramamonjison, Jabo Serge Byusa, Giuseppe Carenini, Zirui Zhou, Yong Zhang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17874
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating LLM Reasoning in the Operations Research Domain with ORQA Mahdi Mostajabdaveh, Timothy T. Yu, Samarendra Chandan Bindu Dash, Rindranirina Ramamonjison, Jabo Serge Byusa, Giuseppe Carenini, Zirui Zhou, Yong Zhang Abstract In this paper, we introduce and apply O perations R esearch Q uestion A nswering (ORQA), a new benchmark, to assess the generalization capabilities of Large Language Models (LLMs) in the specialized technical domain of Operations Research (OR). This benchmark is designed to evaluate whether LLMs can emulate the knowledge and reasoning skills of OR experts when given diverse and complex optimization problems. The dataset, crafted by OR experts, presents real-world optimization problems that require multi-step reasoning to build their mathematical models. Our evaluations of various open-source LLMs, such as LLaMA 3.1, DeepSeek, and Mixtral reveal their modest performance, indicating a gap in their aptitude to generalize to specialized technical domains. This work contributes to the ongoing discourse on LLMs’ generalization capabilities, providing insights for future research in this area. The dataset and evaluation code are publicly available 1 1 1 https://github.com/nl4opt/ORQA . 1 Huawei Technologies Canada, 4321 Still Creek Dr, Burnaby, BC, V5C 6S7, Canada 2 University of Toronto, 40 George St, Toronto, ON, M5S 2E4, Canada 3 University of British Columbia, 2366 Main Mall, Vancouver, BC, V6T 1Z4, Canada {mahdi.mostajabdaveh1, timothy.yu, rindranirina.ramamonjison, jabo.byusa, zirui.zhou, yong.zhang3}@huawei.com, [email protected], [email protected]. Introduction The ability of Large Language Models (LLMs) to follow human instructions and perform diverse tasks has made them an exciting area of investigation. Moreover, the considerable interest in adopting LLMs across various complex technical domains (e.g., medicine ( Gao et al. 2023 ; Zhou et al. 2023 ) ) highlights their potential for significant societal impact. A particularly compelling driver for this adoption is the potential of LLMs to automate many tasks, reducing human intervention and improving productivity. However, as LLMs are becoming integrated into the workflow of various industries, it is important to thoroughly understand their capabilities and limitations ( Khatun and Brown 2024a ; Baktash and Dawodi 2023 ; Truong et al. 2023 ) . Of particular interest is their ability to reason and perform new challenging tasks across different domains, which are reported critical limitations of LLMs ( Arkoudas 2023 ; Shen and Kejriwal 2023 ) . Our work addresses this need by introducing a new benchmark dataset and applying it to assess these limitations. To evaluate an LLM’s ability to generalize to a new domain, we focus on the field of Operations Research (OR). The choice of this domain is deliberate and significant. First, OR is important for making decisions in various industries ( Petropoulos et al. 2024 ) . Second, there are many types of optimization problems applied to real-world applications ranging from production scheduling ( Mostajabdaveh, Salman, and Tahmasbi 2022 ) to creating efficient delivery routes for trucks ( Vidal, Laporte, and Matl 2020 ) . Third, optimization modeling presents a unique challenge due to the expert-level knowledge and reasoning skills it requires ( Hillier and Lieberman 2015 ) , adding a layer of complexity to the automation of this task. Some recent studies also report modest performances of SOTA LLMs such as GPT-4 and Llama2 for optimization model building tasks ( AhmadiTeshnizi, Gao, and Udell 2024 ; Mostajabdaveh et al. 2024 ) . Finally, OR is a niche field with limited publicly available text corpora or optimization model code ( Xiao et al. 2023 ; AhmadiTeshnizi, Gao, and Udell 2024 ) , making it an ideal testbed to assess the generalizability of LLMs, reducing the risk of data contamination. To assess an LLM’s knowledge and reasoning skills on unseen, diverse, and complex optimization problems, we propose ORQA ( O perations R esearch Q uestion A nswering), a new multi-choice Question Answering (QA) benchmark dataset, crafted and verified by OR experts. Each dataset instance (Figure 1 ; left) presents a natural language description of an optimization problem along with a question that requires multi-step reasoning (Figure 1 ; right) to answer correctly. Figure 1: Left : Dataset instance containing the description, question, options, and answer (bold). Right : Example reasoning steps needed to answer the simple question. Our study is significant for several reasons. First, it contributes to the ongoing dialogue on the generalizability of LLMs. Although many benchmarks claim that LLMs can replicate expert-level knowledge across various technical domains, the actual extent of these models’ generalization capabilities remains an open question ( Alzahrani et al. 2024 ) . Our benchmark offers a new perspective by focusing on a specialized technical domain that lacks a large-scale, high-quality dataset. To the best of our knowledge, this is the first multi-choice QA dataset in the field of Operations Research. Second, this study has implications for understanding the potential and limitations of LLMs in automating tasks in niche technical fields like OR, where expert knowledge and complex multi-step reasoning are crucial. Optimization modeling requires access to specialized OR experts, which is often impractical for most potential users due to the associated costs. We choose to focus on translating textual problem descriptions into mathematical optimization models rather than directly solving optimization problems. Even specialized AI models struggle with scalability and generalization when solving simple OR problems ( Joshi et al. 2022 ) . We also highlight the challenges of constructing a technical dataset like ORQA. Samples of optimization problems are inherently complex and require significant time and effort to create and verify. Moreover, ensuring correctness demands annotators with graduate-level education or extensive experience in OR modeling, making the process expensive, time-consuming, and labor-intensive. Related Work LLMs applied to operations research: Within the field of OR, LLMs are being investigated for their ability to formulate optimization models ( Fan et al. 2024 ) . Ramamonjison et al. (2022a) proposed using generative models to automate the formulation of OR problems from natural language. Building on this, Ramamonjison et al. (2022b) introduced methods for recognizing entities and parsing optimization formulations from text. However, these works primarily utilize a toy dataset of elementary linear programming word problems. More recently, AhmadiTeshnizi, Gao, and Udell (2024) proposed the NLP4LP dataset, which includes 332 instances with structured natural language descriptions (SNOP), parameter data values, optimal value, and optimal solution. However, the majority of the problems are still toys, their description is technical, with mathematical notation, and lacks context from real applications. Xiao et al. (2023) introduced the ComplexOR dataset with only 37 optimization problems. This dataset includes NL descriptions, optimization models, and several input data. While their problem descriptions are context-aware, they often mention the related optimization problem by name (e.g., lot-sizing problem with setup), and only cover a narrow range of application domains. With more than 1.5k instances, our dataset is the largest, offering significant value through its depth, rigor, realism, and diversity. Another disadvantage of existing mentioned datasets is they require the optimization model to be solved for their evaluation. LLMs must generate a model code from the NL description, which is then fed to a solver with data to obtain the optimal value or solution. The evaluation focuses solely on the correctness of the optimal value or solution, presenting two key limitations: (1) it is an end-to-end evaluation that cannot differentiate between minor notation errors and entirely incorrect structures; (2) it