Difficulty-Aware Agent Orchestration in LLM-Powered Workflows
Jinwei Su, Yinghui Xia, Qizhen Lan, Xinyuan Song · 2025
Read Difficulty-Aware Agent Orchestration in LLM-Powered Workflows by Jinwei Su, Yinghui Xia, Qizhen Lan, Xinyuan Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-11079.txt.
Introduction
Jinwei Su and eight co-authors present this 2025 artificial intelligence paper on teams of language-model agents that split a question among several specialized workers. Fixed workflows waste effort on easy questions and fall short on hard ones, so the authors let the workflow itself adapt.
Their system, called DAAO, first guesses how hard a query is, then assembles a matching workflow from reusable building blocks and routes each step to a suitable model. Feedback on whether the workflow succeeded keeps the difficulty guesses honest. Across six test benchmarks, the adaptive setup beat earlier multi-agent systems on both accuracy and running cost, giving simple questions lean treatment and hard questions heavier machinery.
Catalog record
| CREATOR | Jinwei Su, Yinghui Xia, Qizhen Lan, Xinyuan Song |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-11079
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Difficulty-Aware Agent Orchestration in LLM-Powered Workflows Jinwei Su, Yinghui Xia, Qizhen Lan, Xinyuan Song Abstract Large Language Model (LLM)-based agentic systems have shown strong capabilities across various tasks. However, existing multi-agent frameworks often rely on static or task-level workflows, which either over-process simple queries or underperform on complex ones, while also neglecting the efficiency-performance trade-offs across heterogeneous LLMs. To address these limitations, we propose Difficulty-Aware Agentic Orchestration (DAAO) , a dynamic framework that adapts workflow depth, operator selection, and LLM assignment based on the difficulty of each input query. DAAO comprises three interdependent modules: a variational autoencoder (VAE) for difficulty estimation, a modular operator allocator, and a cost- and performance-aware LLM router. By leveraging heterogeneous LLMs and dynamically tailoring workflows, DAAO enables fine-grained, query-specific reasoning strategies. DAAO outperforms prior multi-agent systems in both accuracy and inference efficiency across six benchmarks. We will release our code and implementation details upon publication. Figure 1: The overall framework of our proposed DAAO. Introduction Large Language Model (LLM)-based agents ( Richards and et al. 2023 ; Nakajima 2023 ; Reworkd 2023 ) have exhibited remarkable capabilities across a wide spectrum of tasks, including question answering ( Zhu et al. 2024 ) , data analysis ( Hong et al. 2024 ; Li et al. 2024 ) , decision-making ( Song et al. 2023 ) , code generation ( Shinn, Labash, and Gopinath 2023 ) and web navigation ( Deng et al. 2024 ) . Building upon the success of single agents, recent advancements reveal that organizing multiple LLM-based agents into structured agentic workflows can significantly enhance task performance. In such workflows, agents can interact either cooperatively ( Zhuge et al. 2024 ) or competitively ( Zhao et al. 2023 ) depending on the task context. These multi-agent systems can overcome the cognitive and functional limitations of individual models ( Du et al. 2023 ; Liang et al. 2023 ; Wang et al. 2023b ; Jiang, Ren, and Lin 2023 ; Wu et al. 2023 ; Zhang et al. 2024a ) , thereby exhibiting collective intelligence similar to human collaboration in a society of agents. In recent years, the research community has focused on automating multi-agent system design. For instance, DsPy ( Khattab et al. 2023 ) and EvoPrompting ( Guo et al. 2023 ) automate prompt optimization, GPTSwarm ( Zhuge et al. 2024 ) optimizing inter-agent communication, and EvoAgent ( Yuan et al. 2024 ) self-evolving agent profiling. However, these systems are often constrained by limited search spaces and rigid representation paradigms, resulting in marginal performance gains and limited adaptability to diverse task requirements. Subsequently, ADAS ( Hu, Lu, and Clune 2024 ) and AFlow ( Zhang et al. 2024b ) employ code as representation for workflow, facilitating robust and flexible workflow searches through different paradigms, with ADAS utilizing heuristic search and AFlow adopting Monte Carlo tree search. MaAS ( Zhang et al. 2025 ) proposes an agentic supernet to generate a query-specific multi-agent system for each user query. Despite their success, existing automation pipelines often lack complex adaptivity and LLM heterogeneity. Task-level workflows ( Hu, Lu, and Clune 2024 ; Zhang et al. 2024b ) are typically built as uniform multi-agent systems for entire task categories, achieving strong metrics like accuracy and pass@k but relying on heavy pipelines with excessive LLM calls and tool usage. This design over-processes simple queries, wasting resources and overlooking factors like token cost and latency. Query-level workflows ( Zhang et al. 2025 ) introduce input-specific adaptation, but their granularity is often insufficient, leading to suboptimal or oversimplified workflows for difficult inputs. These limitations motivate a difficulty-adaptive framework that dynamically balances complexity and cost. Furthermore, most workflows ( Hu, Lu, and Clune 2024 ; Zhang et al. 2024b ; Zhang et al. 2025 ) rely on a single large LLM (e.g., GPT-4o ( OpenAI 2024 ) ), ignoring recent findings ( Ye et al. 2025 ; Chen, Zaharia, and Zou 2023 ; Hu et al. 2024 ) that different LLMs exhibit complementary capabilities. Smaller models can outperform larger ones on specific tasks while significantly reducing cost. Thus, there is increasing support for agentic workflows that integrate diverse LLMs of varying sizes, leveraging their specialization to improve performance and efficiency. To address the challenges of adapting reasoning strategies to queries with varying difficulty and domain characteristics, we propose Difficulty-Aware Agentic Orchestration (DAAO) . DAAO is a dynamic framework that estimates the difficulty of each incoming query and composes optimized workflows by selecting suitable agentic operators—such as Chain-of-Thought (CoT) ( Wei et al. 2022 ) , Multi-Agent Debate ( Du et al. 2023 ) , and ReAct ( Yao et al. 2023 ) —and assigning them to appropriate LLMs. By tailoring both workflow depth and model assignment based on query characteristics, DAAO balances reasoning effectiveness and computational cost. Furthermore, the system refines its predictions over time through feedback, enabling continual adaptation. Technically, DAAO decomposes the orchestration process into three interdependent modules. It begins with a query difficulty estimator , implemented as a variational autoencoder (VAE) ( Kingma and Welling 2014 ) , which encodes each input query into a latent representation reflecting its difficulty. This difficulty signal then guides the agentic operator allocator , which determines the depth of the workflow and selects suitable reasoning operators based on both the query and its inferred complexity. Finally, the LLM router assigns each selected operator to a language model that best fits its role, enabling heterogeneous and cost-effective reasoning. Together, these modules form a cohesive and adaptive orchestration framework that supports flexible, query-specific agentic workflows. We conduct comprehensive evaluations on six widely adopted benchmarks. Empirical results reveal that our DAAO achieves both state-of-the-art performance and remarkable cost-reduction. For instance, our DAAO surpasses state-of-the-art automated multi-agent systems by 11.21\% in accuracy, while requiring only 64\% of their inference costs, owing to its difficulty-aware routing and adaptive workflow construction. Our key contributions are as follows: • Query-Level Difficulty Estimation for Workflow Adaptation: We propose a difficulty-aware orchestration framework that estimates the complexity of each input query using a variational autoencoder (VAE). This latent difficulty signal informs both the depth and structure of the workflow, enabling fine-grained, query-specific adaptation beyond static or task-level designs. • Modular Agentic Workflow Construction with Heterogeneous Operators and LLMs: We develop a modular pipeline that selects agentic operators based on query difficulty and context, and assigns each to the most suitable LLM. This fine-grained routing leverages the complementary strengths of diverse models to enhance reasoning performance while reducing cost. • Efficient and Scalable Execution with Empirical Gains: Our method achieves state-of-the-art performance on six benchmarks, outperforming prior multi-agent systems by up to 11.21% in accuracy while using only 64% of the inference cost. The effectiveness of difficulty-guided orchestration is further validated through comprehensive ablation and cost-efficiency analysis. Figure 2: Case study and visualization for MaAS ( Zhang et al. 2025 ) and Ours. Queries are from MATH, HumanEval and GAIA benchmarks. Left: our method solves the query while MaAS fails. Middle: our workflow is simpler for easy queries. Right: our method leverages heterogeneous LLMs within the same workflow structure. Related Work Automated Agentic Workflows. The development of agentic workflows has evolved from manual configurations to automated systems, with the latter offering improved