Aligning Compound AI Systems via System-level DPO

Xiangwen Wang, Yibo Jacky Zhang, Zhoujie Ding, Katherine Tsai, Haolun Wu, Sanmi Koyejo · 2025

Read Aligning Compound AI Systems via System-level DPO by Xiangwen Wang, Yibo Jacky Zhang, Zhoujie Ding, Katherine Tsai, Haolun Wu, Sanmi Koyejo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17721.txt.

Introduction

Xiangwen Wang and co-authors align compound AI systems with human preferences in a 2025 paper. Such systems chain models and tools together and beat single models on many tasks, but tuning the whole chain is hard. Parts interact in ways that block standard gradient methods, and a judgment on the whole system does not split neatly into judgments on parts.

The authors draw the system as a flow graph of parts and data, then extend a known preference-tuning method to the whole graph at once. Two variants cover cases with or without a ready preference data set for the system. Tests on a paired language and image system and on a team of language models showed the approach works in practice.

Catalog record

CREATORXiangwen Wang, Yibo Jacky Zhang, Zhoujie Ding, Katherine Tsai, Haolun Wu, Sanmi Koyejo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-17721

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Aligning Compound AI Systems via System-level DPO Xiangwen Wang, Yibo Jacky Zhang, Zhoujie Ding, Katherine Tsai, Haolun Wu, Sanmi Koyejo Abstract Compound AI systems, comprising multiple interacting components such as LLMs, foundation models, and external tools, have demonstrated remarkable improvements compared to single models in various tasks. To ensure their effective deployment in real-world applications, aligning these systems with human preferences is crucial. However, aligning the compound system via policy optimization, unlike the alignment of a single model, is challenging for two main reasons: (i) non-differentiable interactions between components make end-to-end gradient-based optimization method inapplicable, and (ii) system-level preferences cannot be directly transformed into component-level preferences. To address these challenges, we first formulate compound AI systems as Directed Acyclic Graphs (DAGs), explicitly modeling both component interactions and the associated data flows. Building on this formulation, we introduce SysDPO , a framework that extends Direct Preference Optimization (DPO) to enable joint system-level alignment. We propose two variants, SysDPO-Direct and SysDPO-Sampling , tailored for scenarios depending on whether we construct a system-specific preference dataset. We empirically demonstrate the effectiveness of our approach across two applications: the joint alignment of a language model and a diffusion model, and the joint alignment of an LLM collaboration system. 1 Introduction Compound AI systems, which consist of multiple interacting AI components, serve as promising frameworks to push beyond the model capabilities and achieve state-of-the-art performance zaharia2024shift ; chen2024are ; kandogan2024blueprint ; lin2024llm . For example, ChatGPT integrates a Large Language Model (LLM), a DALL-E image generator, a web browser plugin, and various other system components to support diverse user needs achiam2023gpt . A multi-agent system consisting of multiple LLMs working collaboratively achieves improved performance compared to a single agent wang2024mixture . A Retrieval-Augmented Generation (RAG) system combines large language models with information retrieval capabilities and is capable of answering time-sensitive queries. A multi-LLM routing system includes a router that dynamically selects among a diverse set of models according to user queries to maximize the model performance hu2024routerbench . These examples illustrate how compound AI systems leverage LLMs alongside complementary modules to tackle complex tasks beyond the reach of a single LLM. Ensuring effective collaboration among components is crucial for compound AI systems to function reliably. It also plays a critical role in aligning the outputs of the system with human preferences and in ensuring the safety and ethical standards lin2024llm . However, simply integrating multiple models does not guarantee effective coordination, as illustrated by a failure case involving an LLM (GPT-4) and a diffusion model (DALL·E), shown in Figure 1 . Our experiments (Section 5 ) reveal that an instructed tuned Llama-3-8B combined with Stable Diffusion XL achieve a correctness rate of only 32% on similar tasks. These failure cases highlight the critical need to develop a new framework to align compound AI systems. Calm Cat Slightly Irritated Cat Very Angry Cat (a) Figure 1: The figure illustrates the challenges in a compound system composed of the GPT-4 and the image generator DALL-E. Given the user prompt to GPT-4, “ Generate three separate images of a cat being progressively angrier ”, (a) shows the results from one query, and (b) represents the results from another query. The captions under each image summarize the prompts generated by GPT-4 for DALL-E (complete prompts in Appendix A ), where prompts from both queries reflect progressions in anger. Similarly, DALL-E accurately generates the images following the given prompts. However, (a) fails to demonstrate a clear visual progression of anger compared to (b) , highlighting GPT-4’s inconsistent collaboration with DALL-E. Slightly Annoyed Cat Angry Cat Furious Cat (b) While alignment techniques for monolithic models are well-studied rafailov2024direct ; ziegler2019fine ; bai2022training , aligning compound systems remains an open challenge. Standard methods such as Direct Preference Optimization (DPO) rafailov2024direct and Reinforcement Learning from Human Feedback (RLHF) ouyang2022training ; ziegler2019fine ; bai2022training are not directly applicable to compound systems for three primary bottlenecks: (i) Non-differentiable interactions : Components in a compound system often interact through non-differentiable channels, such as natural language, preventing end-to-end optimization and making credit assignment across components difficult. (ii) Non-decomposable preferences: Aligning each component independently is inadequate, as system-level preferences are not simply decomposable into individual preferences. Moreover, effective coordination between components is essential but cannot be captured through isolated alignment. (iii) Lack of fine-grained benchmarks: Most alignment benchmarks are constructed to evaluate the entire system; benchmarks for individual sub-tasks might not exist. In light of these challenges, there is an urgent need to develop methodologies for aligning compound AI systems. While recent studies have explored prompting techniques and instruction tuning approaches yuksekgonul2024textgrad ; lin2024llm ; shinn2024reflexion , we take an alternative approach to address these challenges. Our main contributions are summarized below: • We model compound AI systems as Directed Acyclic Graphs and propose SysDPO , a Direct Preference Optimization (DPO)-based alignment framework with two variants— SysDPO-Direct and SysDPO-Sampling —for settings with or without system-specific datasets (Section 2 ). • We provide a theoretical analysis showing that SysDPO achieves \beta -perfect alignment under the population setting, generalizing standard DPO guarantees to compound systems (Section 3 ). • We demonstrate SysDPO with two applications: aligning an LLM and a text-to-image diffusion model, as well as aligning two LLMs (Section 4 ). Our experiment results indicate that aligning compound AI systems increases the success rate in handling complex instructions (Section 5 ). These results deepen our understanding of alignment challenges in compound AI systems and provide a foundation for future research. 2 The SysDPO Framework In this section, we introduce the framework of SysDPO through intuitive motivations, where the theoretical justification is provided later in Section 3 . We start by reviewing prior work on Bradley-Terry model bradley1952rank and DPO rafailov2024direct , then move to introduce the SysDPO pipeline. Bradley-Terry (BT) model. Given an input x , the system generates two pairs of outputs z,z^{\prime} . We represent the preference of the outputs as (z\succ z^{\prime}\mid x) if z is preferred over z^{\prime} by a preference oracle, e.g., human labelers. To model the preference, we use the Bradley-Terry model – a common preference model used in alignment rafailov2024direct ; stiennon2020learning ; bai2022training , that represents the preference distribution as \displaystyle\pref(z\succ z^{\prime}\mid x)=\frac{\exp(r^{*}(x,z))}{\exp(r^{*}(x,z))+\exp(r^{*}(x,z^{\prime}))}\in(0,1), (2) where r^{*}(x,z) is the ground truth reward model. Drawing preference from the preference oracle, the winning sample is assigned to z^{w}\leftarrow z and the losing sample is assigned to z^{l}\leftarrow z^{\prime} with probability \pref(z\succ z^{\prime}\mid x) . Using the preference oracle, one can construct a preference distribution {\mathcal{D}} composed of preference pairs (x,z^{w},z^{l}) . Direct Preference Optimization. DPO rafailov2024direct aligns the model \theta using the preference distribution {\mathcal{D}} by minimizing the following loss: \displaystyle L(\theta)=-\mathbb{E}_{(x,z^{w},z^{l})\sim{\mathcal{D}}}\left[\log\sigma\left(\beta\log\frac{p_{\theta}(z^{w}|x)}{p_{\bar{\theta}}(z^{w}|x)}-\beta\log\frac{p_{\theta}(z^{l}|x)}{p_{\bar{\theta}}(z^{l}|x)}\right)\right], (3) where \bar{\theta} denotes the reference model of \theta , \sigma(\cdot) stands for the sigmoid function. When it comes to compound AI systems, the optimization challenge arises from the structure of \theta , which represents a collection of model parameters. These models may communicate in complex ways that are non-differentiable, e.g., by exchanging plain text or task-specific outputs. Moreover, for a compound system with intermediate generations y , the system’s generation probability takes the form of an integral p_{\theta}(z\mid x)=\int p_{\theta}(z,y\mid x)\mathop{}\!\mathrm{d}y , raising challenges for optimization. Therefore, the alignment of compound AI

Read the complete text