MSCoRe: A Benchmark for Multi-Stage Collaborative Reasoning in LLM Agents
Yuzhen Lei, Hongbin Xie, Jiaxing Zhao, Shuangxue Liu, Xuan Song · 2025
Read MSCoRe: A Benchmark for Multi-Stage Collaborative Reasoning in LLM Agents by Yuzhen Lei, Hongbin Xie, Jiaxing Zhao, Shuangxue Liu, Xuan Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17628.txt.
Introduction
Yuzhen Lei, Hongbin Xie, Jiaxing Zhao, Shuangxue Liu, and Xuan Song built a benchmark for agents that must reason in stages. Their 2025 arXiv paper notes that language models shine at single-domain questions, yet real work chains tasks across stages. Current benchmarks test isolated skills and miss how agents coordinate and optimize over a long job with no outside guidance.
Their benchmark, MSCoRe, holds a large set of domain question-answer items from the automotive, pharmaceutical, electronics, and energy sectors. A three-phase pipeline samples items, generates pairs, and grades quality at several levels. Tasks sort into three difficulty tiers by stage coverage and complexity. Tests of leading agents show commercial models ahead on all fronts. A clear gap in matching scores still splits simple tasks from complex ones, which is the headroom the benchmark was built to expose.
Catalog record
| CREATOR | Yuzhen Lei, Hongbin Xie, Jiaxing Zhao, Shuangxue Liu, Xuan Song |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17628
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MSCoRe: A Benchmark for Multi-Stage Collaborative Reasoning in LLM Agents Yuzhen Lei, Hongbin Xie, Jiaxing Zhao, Shuangxue Liu, Xuan Song Abstract Large Language Models (LLMs) have excelled in question-answering (QA) tasks within single domains. However, their reasoning and coordination capabilities in complex, multi-stage scenarios remain underexplored. Existing benchmarks typically focus on isolated tasks or narrow domains, overlooking models’ abilities for multi-stage collaboration and optimization without explicit external guidance. To bridge this gap, we propose MSCoRe , a novel benchmark comprising 126696 domain-specific QA instances spanning scenarios in automotive, pharmaceutical, electronics, and energy sectors. The dataset is created using a structured three-phase pipeline: dynamic sampling, iterative question-answer generation, and a multi-level quality assessment to ensure data quality. Tasks are further categorized into three difficulty levels according to stage coverage and complexity. With MSCoRe, we have conducted a comprehensive evaluation of various state-of-the-art LLM agents. The commercial models performed best across all tasks and scenarios, but a notable gap in ROUGE scores remains between simple and complex tasks. We also tested the models’ robustness and found that their performance is negatively affected by noisy data. MSCoRe provides a valuable new resource for the community to evaluate and improve multi-stage reasoning in LLM agents. The code and data are available at https://github.com/D3E0-source/MSCoRE . 1 Introduction Figure 1: An illustrative multi-stage QA example highlighting design–manufacturing synergy for vehicle fuel efficiency improvement in MSCoRe Open-domain question answering (QA) serves as a cornerstone task for evaluating advanced capabilities of Artificial Intelligence(AI) language models [ 1 ] . It requires both precise intent understanding and the generation of coherent, logically sound responses. A rich ecosystem of benchmarks has been developed to this end. General-purpose benchmarks such as WikiQA [ 2 ] , MMLU [ 3 ] , CommonsenseQA [ 4 ] , and ZeroShot [ 5 ] span diverse domains and provide rigorous evaluation of models’ foundational reasoning and knowledge. Currently, domain-specific datasets such as EQUALS [ 6 ] for law, DISC-FinLLM [ 7 ] for finance, and MedExQA [ 8 ] for medicine have emerged to assess specialized expertise. However, a critical limitation persists across these evaluations: they predominantly treat tasks as isolated, single-stage problems. This approach assesses knowledge within a specific context, but overlooks the crucial interdependencies and collaborative reasoning required across different operational stages within a complex domain. This limitation is particularly salient in real-world industrial applications where workflows are inherently multi-staged. For example, the automotive value chain involves interconnected stages from design and manufacturing to supply chain and quality inspection [ 9 ] . A design choice directly impacts manufacturing feasibility, while a supply chain disruption can halt production. Similarly, in the pharmaceutical industry, quality inspection results provide essential feedback for optimizing production processes [ 10 ] . Effectively navigating these environments demands more than isolated knowledge; it requires the ability to perform collaborative reasoning that considers downstream consequences and upstream constraints. Existing evaluation frameworks are ill-equipped to measure this vital, holistic problem-solving capability. To bridge this crucial evaluation gap, we introduce MSCoRe, a novel benchmark designed to assess the multi-stage collaborative reasoning of LLMs. Different from existing benchmarks, MSCoRe presents complex scenarios where a viable solution requires synthesizing information and considering trade-offs across multiple, interdependent stages (as shown in Figure 1 ). The benchmark comprises 1080,00 high-quality QA pairs spanning four intricate domains: automotive, pharmaceutical, e-commerce, and energy value chain(as shown in Figure 2 ). We constructed this dataset using a systematic pipeline featuring high-quality seed data, refined prompt engineering, and a multi-level quality control mechanism with feedback-driven optimization. To enable a fine-grained analysis, we stratify the tasks into three difficulty levels—simple, medium, and difficult—based on their complexity and the number of reasoning stages involved. With MSCoRe, we have conducted a comprehensive evaluation of various state-of-the-art LLMs, including the commercial models like GPT-4o [ 11 ] , GPT-3.5-turbo, and Claude-3.5-haiku, as well as open-source models like Deepseek-R1 [ 12 ] , and Qwen series [ 13 ] . The results indicate that even models excelling in simple tasks face significant challenges when tackling full-chain tasks. Furthermore, we introduced various forms of noise—including unstandardized formats, incomplete information, and semantic inaccuracies—into selected sample datasets to assess models’ robustness and interference resistance under adverse input conditions. In summary, this work makes the following contributions: • We propose a systematic automated framework to construct MSCoRe, a large-scale benchmark with 126696 multi-stage QA instances across four industrial domains. The framework incorporates dynamic sampling, iterative generation, and multi-level quality control, and its effectiveness is validated through a Turing test in which experts misclassified over 85% of AI-generated data as human-authored, confirming its human-level quality. • We conduct comprehensive experiments on 15 state-of-the-art LLMs, establishing rigorous baselines and revealing critical findings: commercial models achieve the strongest performance but still show substantial degradation from single-stage to full-chain tasks, exhibit marked brittleness under noisy or incomplete inputs, and display mixed sensitivity to few-shot examples depending on model scale and capability. • We provide a foundation for future research by releasing MSCoRe as a challenging testbed and offering methodological insights that highlight key limitations of current LLM agents, guiding the development of more robust, adaptive, and practically deployable systems for complex multi-stage reasoning tasks. Figure 2: Data distribution across various industry sectors and workflow stages. 2 Related Work 2.1 Benchmarks for General LLM capabilities The rapid evolution of Large Language Models (LLMs) has been accompanied by the development of comprehensive benchmarks designed to evaluate their core capabilities. Early general benchmarks like GLUE [ 14 ] and SuperGLUE [ 15 ] established a foundation for evaluating natural language understanding. However, as LLMs grew in scale and complexity, more extensive evaluation suites became necessary. Benchmarks such as MMLU assess knowledge across a massive range of subjects, while BIG-bench [ 16 ] tests for a diverse set of emergent abilities. More recent efforts like HELM [ 17 ] advocate for a multi-metric, holistic evaluation to provide a more complete picture of model performance, covering aspects like accuracy, fairness, and efficiency. Lately, the evaluation focus has shifted towards more interactive and agent-like behaviors. For instance, AgentBench [ 18 ] evaluates LLMs in complex, multi-turn open-ended environments, and ToolBench [ 19 ] assesses their ability to use external tools to solve problems. While these benchmarks are crucial for understanding the general reasoning and task-execution abilities of LLMs, they typically consist of a collection of discrete tasks. They often do not capture the intricate, causal dependencies inherent in a single, coherent industrial workflow, where the output of one stage becomes a critical and constraining input for the next. MSCoRe is specifically designed to fill this gap by modeling these inter-stage relationships. 2.2 Domain-Specific LLM Evaluation Recognizing that real-world applications require specialized knowledge, there has been a significant trend towards creating domain-specific benchmarks. In high-stakes fields like medicine, benchmarks such as MedQA [ 20 ] and MedMCQA [ 8 ] evaluate clinical knowledge by testing models on medical exam questions. The recent evaluation framework for Med-PaLM 2 further highlighted the need for rigorous assessment against real-world medical challenges [ 21 ] . Similarly, the finance sector has seen the introduction of benchmarks like FinQA [ 6 ] and the more recent FinEval [ 22 ] , which focus on