Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale

Jiale Liu, Yifan Zeng, Shaokun Zhang, Chi Zhang, Malte Højmark-Bertelsen, Marie Normann Gadeberg, Huazheng Wang, Qingyun Wu · 2025

Read Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale by Jiale Liu, Yifan Zeng, Shaokun Zhang, Chi Zhang, Malte Højmark-Bertelsen, Marie Normann Gadeberg, Huazheng Wang, Qingyun Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03973.txt.

Introduction

Jiale Liu and seven co-authors scale up the tuning of AI agents by splitting the job into pieces. A common trick is to let a language model improve an agent by reading its past runs, but big data sets overflow the model's window and blur the patterns. Their 2025 framework, FGO, divides the runs into small subsets. Each subset is optimized on its own, and the improved parts are then merged step by step into one agent.

Tests on a household task benchmark and other suites check whether the merged agent keeps the gains. Progressive merging is the safeguard against fixes that clash. The authors aim at real agent programs, whose logs grow far past any single prompt. Their method treats optimization as an engineering pipeline, not a single clever prompt.

Catalog record

CREATORJiale Liu, Yifan Zeng, Shaokun Zhang, Chi Zhang, Malte Højmark-Bertelsen, Marie Normann Gadeberg, Huazheng Wang, Qingyun Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03973

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale Jiale Liu, Yifan Zeng, Shaokun Zhang, Chi Zhang, Malte Højmark-Bertelsen, Marie Normann Gadeberg, Huazheng Wang, Qingyun Wu Abstract LLM-based optimization has shown remarkable potential in enhancing agentic systems. However, the conventional approach of prompting LLM optimizer with the whole training trajectories on training dataset in a single pass becomes untenable as datasets grow, leading to context window overflow and degraded pattern recognition. To address these challenges, we propose Fine-Grained Optimization (FGO), a scalable framework that divides large optimization tasks into manageable subsets, performs targeted optimizations, and systematically combines optimized components through progressive merging. Evaluation across ALFWorld, LogisticsQA, and GAIA benchmarks demonstrate that FGO outperforms existing approaches by 1.6-8.6% while reducing average prompt token consumption by 56.3%. Our framework provides a practical solution for scaling up LLM-based optimization of increasingly sophisticated agent systems. Further analysis demonstrates that FGO achieves the most consistent performance gain in all training dataset sizes, showcasing its scalability and efficiency. 1 Introduction Large Language Models (LLMs) have emerged as powerful optimizers for LLM systems, capable of analyzing execution trajectories and refining system modules like prompts ( Yang et al. 2024 ; Zhou et al. 2022 ; Khattab et al. 2023 ; Opsahl-Ong et al. 2024 ) , tools ( Qian et al. 2023 ; Zhang et al. 2024c ; Zhang et al. 2024b ; Wang et al. 2024a ) . These agentic systems have shown promising results in enhancing agent performance across various domains, including reasoning ( Cheng et al. 2024 ; Zelikman et al. 2023 ) , software engineering ( Jimenez et al. 2023 ; Pan et al. 2024 ) , data analysis ( Hu et al. 2024b ; Jing et al. 2024 ) , computer using ( Wang et al. 2025 ; Xie et al. 2025 ; Abuelsaad et al. 2024 ; Xia and Luo 2025 ) . Figure 1: Agent optimization approaches. (a) Basic agent execution process. (b) Traditional all-at-once optimization faces context overflow and inferior performance with large trajectory data. (c) Our method: divide-and-conquer optimization with progressive merging enables scalable processing of large datasets. However, due to the increasing volume of data required for optimizing LLM agentic systems autonomously, directly applying LLM-based optimization approaches encounters a fundamental scalability issue. Existing methods typically concatenate all execution trajectories on the training data and perform optimization in an all-at-once manner, feeding the entire dataset into the LLM optimizer in a single prompt. While this approach works for optimization tasks with small-scale data, it becomes problematic as the data grows. For instance, in the GAIA benchmark ( Mialon et al. 2023 ) , agents normally rely on external tools to collect real-world information and generate lengthy execution traces for subsequent optimization, which is filled with raw documents and complex intermediate reasoning steps, even challenge for human to parse. This increasing complexity leads to two critical limitations: (1) The concatenated trajectories exceed LLM context windows, forcing truncation of valuable optimization signals. (2) Even when content fits within context windows, LLMs struggle with analyzing long-range dependencies in extensive corpus ( Bai et al. 2024 ; Liu et al. 2024 ; Ni et al. 2024 ; Ravaut et al. 2024 ; Li et al. 2024 ) , making it hard for the LLM optimizer to capture subtle patterns and relationships between execution traces. As a result, such approaches can produce suboptimal solutions, particularly in complex scenarios where understanding the intricate relationships between different execution trajectories is crucial for improving agent performance. To address these scalability challenges, we introduce FGO, a framework that enables efficient optimization of LLM-based agentic systems with large-scale data. Specifically, FGO operates through three components: (1) Task division that breaks down the large training dataset into more manageable subsets, (2) Fine-grained optimization enabling efficient processing of each subset, and (3) Progressive module merging that adaptively combines optimized modules while preserving crucial insights from each subset. This design allows FGO to effectively handle larger optimization tasks while maintaining high-quality results. We evaluate FGO by optimizing two agent modules: instruction prompts and tools agent could access. Across diverse tasks including ALFWorld ( Shridhar et al. 2020 ) , LogisticsQA, and GAIA ( Mialon et al. 2023 ) . Agent trained with FGO produces significant performance gains across all datasets, ranging from 8.3% to 38.1%, outperforming other optimization methods by 1.6%-8.6%. Further analysis reveals that FGO maintains superior performance across varying training dataset sizes, highlighting its scalability and stability. Notably, FGO achieves these improvements while reducing prompt token consumption by 56.3% and increasing optimization efficiency by 7.6% compared to conventional all-at-once optimization. Our contributions are threefold: (1) We identify and analyze the scalability limitations in current LLM-based optimization approaches for agentic systems. (2) To address the scalability limitation, we propose FGO, a scalable optimization framework that effectively handles large-scale agent optimization through task division and progressive merging. (3) We demonstrate FGO’s effectiveness across diverse tasks and provide insights into its scalability advantages through comprehensive empirical analysis. 2 Preliminary 2.1 Problem Setup LLM Agent Optimizable Modules Agentic systems exhibit complex behavioral patterns emerging from multiple factors. A critical insight in designing such systems lies in the decomposition of the agent’s parameter space into modules that can be independently optimized ( Anthropic 2024a ) . This decomposition enables targeted optimization of specific functional aspects while maintaining global system coherence. Denote the parameter space of agentic system as \Theta , which partitions into trainable modules \{\Theta_{i}\}_{i=1}^{n} governing distinct behavioral dimensions. Each module must satisfy two key properties to qualify as a modular unit. First, the trainability property requires that each module can meaningfully influence the agent’s policy gradients when exposed to specific queries. This ensures the module is sufficiently responsive to reward signals during optimization. Second, the orthogonality property mandates that parameter gradients across different modules exhibit minimal directional alignment during optimization. Such orthogonality constraint ensures modules encode non-redundant functionalities while guaranteeing each contributes uniquely to performance optimization. Agentic System Optimization An agent interacts with an environment \mathcal{E} by generating a sequence of actions in response to a query. Given parameters \theta , the agent’s policy determines actions based on the current state of interaction and observation. These actions along with the observations form a trajectory \tau that represents the agent’s solution attempt for the given query. \begin{gathered}a_{t}\sim\pi(\cdot|a_{1:t},o_{1:t};\theta),\;o_{t+1}\sim\mathcal{E}(\cdot|a_{t}),\;\forall t\in[T]\\ \tau=\mathcal{A}(q;\theta)=(o_{1},a_{1},...,o_{T},a_{T})\end{gathered} (1) The performance is quantified through a loss function \mathcal{L} . Given a distribution \mathcal{D} over query-label pairs (q,y) , we aim to find optimal agent parameters that minimize expected loss across the task distribution. The optimization objective is: \theta^{*}=\arg\min_{\theta\in\Theta}\mathbb{E}_{(q,y)\sim\mathcal{D}}\left[\mathcal{L}(\mathcal{A}(q;\theta),y)\right] (2) This formulation of optimization via tuning modules provides a unified abstraction for analyzing performance-critical factors in agentic system design. In practice, the modules include prompt for task handling ( Wen et al. 2024 ; Wu et al. 2024c ) , long term memory ( Zhang et al. 2024e ) , the available toolbox ( Zhang et al. 2024c ) , the weights of backbone LLM ( Zeng et al. 2024 ; Ma et al. 2024 ) . 2.2 Motivation In LLM-as-optimizer setting, we assume the numeric value of the policy gradient is not accessible in Eq. 2 . This constraint emerges from a practical reality in modern LLM agent systems - the increasing

Read the complete text