CoFi-PGMA: Counterfactual Policy Gradients under Filtered Feedback for Multi-Agent LLMs

Stela Tong, Elai Ben-Gal · 2026

Read CoFi-PGMA: Counterfactual Policy Gradients under Filtered Feedback for Multi-Agent LLMs by Stela Tong, Elai Ben-Gal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22785.txt.

Introduction

Ben-Gal and Tong argue that shared rewards mislead training in multi-agent language-model systems. One reward cannot tell which agent truly caused it, and no single score per agent can capture joint performance once agents interact. They define a background-dependent marginal contribution measure and derive training signals from it, identified through filtered feedback plus a limited budget of exact counterfactual checks. Built into GRPO, their signal lifts routed GSM8K accuracy over winner-take-all training without extra generation cost.

Catalog record

CREATORStela Tong, Elai Ben-Gal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56378
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22785

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoFi-PGMA: Counterfactual Policy Gradients under Filtered Feedback for Multi-Agent LLMs Stela Tong, Elai Ben-Gal Abstract Large language model (LLM) deployments increasingly rely on multi-agent architectures in which multiple models either compete through routing mechanisms or collaborate to produce a final answer. In both settings, the learning signal received by each agent is filtered by the system mechanism. Routing produces selection-gated feedback where only the chosen response is evaluated, while collaboration produces shared rewards that obscure the individual contribution of each agent. As a result, standard RLHF objectives designed for a single deployed policy become misspecified. We introduce CoFi-PGMA (Counterfactual Policy Gradients under Filtered Feedback for Multi-Agent LLMs), a unified framework for learning under filtered feedback in multi-agent LLM systems. Our approach derives a counterfactual per-agent training objective based on marginal contribution, which corrects the learning signal under both routing and collaborative mechanisms. For routing systems, the objective corresponds to off-policy corrections for selection-gated feedback, while for collaborative systems it reduces to leave-one-out difference rewards for credit assignment. We further analyze how softmax routing induces risk-sensitive incentives and provide practical training algorithms that integrate counterfactual estimators, multiturn-aware rewards, and policy optimization methods, and demonstrate the approach on a real-world reasoning dataset. 1 Introduction Many deployed LLM products are no longer a single monolithic policy. Instead, they are multi-agent systems in which multiple specialized models are invoked and a system-level mechanism decides what the user ultimately sees. Two common industrial patterns are (i) routing , where a controller selects among experts (e.g., choosing between different specialist models) and only the selected response is delivered, and (ii) collaboration , where roles such as planner–solver–critic–editor jointly produce one final answer via a composition rule. In both patterns the learning signal received by any single agent is filtered by the mechanism: under routing, reward is observed only for the selected output (selection-gated feedback); under collaboration, a single final reward is shared across agents, obscuring individual contributions (credit assignment). Standard reinforcement learning from human feedback (RLHF) objectives are designed for a single deployed policy receiving direct feedback. In multi-agent systems, however, the system mechanism induces filtered feedback that makes naive per-agent RLHF either statistically biased (because unselected actions are never evaluated) or strategically misaligned (because shared reward does not reflect each agent’s marginal contribution). Our main novelty lies in introducing Counterfactual Policy Gradients under Filtered Feedback for Multi-Agent LLMs (CoFi-PGMA), a unified framework for learning under filtered feedback in multi-agent LLM systems. We instantiate this framework through two concrete mechanisms, competitive routing and collaborative generation, and evaluate the approach on a real-world reasoning dataset. Concretely, we train each agent using a counterfactual advantage signal that measures its marginal contribution to the final outcome, defined as the difference between the observed system reward and the reward that would have been obtained if the agent had not participated. In routing systems, this corresponds to removing the agent from the candidate set; in collaborative systems, it corresponds to replacing the agent’s contribution with a baseline. We instantiate this objective using practical estimators—doubly robust estimators for selection-gated routing and leave-one-out estimators for collaboration—together with multiturn-aware rewards and GRPO optimization. Although motivated by RLHF systems, the framework applies more broadly to multi-agent reinforcement learning settings where feedback is filtered by a system-level mechanism such as routing or aggregation. 2 Related work RLHF RLHF has become a dominant paradigm for aligning large language models with human preferences by learning reward models from comparison data and optimizing policies accordingly ( ouyang2022training ) . Recent methods explore improved optimization procedures for large-scale LLM alignment, such as GRPO, which stabilizes policy updates during training ( shao2024deepseekmath ) . Other work studies reward formulations that capture longer-horizon conversational effects, including multiturn-aware rewards that evaluate the downstream impact of intermediate responses in dialogue systems ( wu2025collabllm ) . These methods provide effective tools for aligning LLM behavior with human preferences. However, most formulations assume a single deployed policy and do not account for the filtered feedback mechanisms that arise in multi-agent LLM systems. Multi-agent RL The multi-agent reinforcement learning literature studies the problem of credit assignment in cooperative systems, where a global reward must be decomposed into meaningful learning signals for individual agents. Prior work proposes mechanisms such as difference rewards and counterfactual baselines that estimate each agent’s marginal contribution to overall system performance ( tumer2000collective ; foerster2018counterfactual ) . These ideas provide principled tools for attributing outcomes in multi-agent settings but are mathematically limited to simplified environments and do not capture more general settings with filtered feedback. Contextual Bandits and Routing Routing mechanisms used in multi-agent LLM systems resemble contextual bandit settings with selection-gated feedback, where only the reward of the chosen action is observed. This partial-feedback setting has been extensively studied in off-policy evaluation and learning from logged bandit data, where techniques such as importance sampling and doubly robust estimators are used to correct for biased feedback logs ( dudik2011doubly ) . Our work connects these strands by studying learning in multi-agent LLM systems under filtered feedback and deriving counterfactual per-agent training objectives that correct the learning signal in both routing and collaborative settings. 3 Framework for RLHF under Filtered Feedback We introduce a unified framework for RLHF in multi-agent systems with filtered feedback. The framework formalizes both routing and collaboration mechanisms and defines counterfactual marginal contribution as the correct learning signal. 3.1 Interaction model We model a conversation as an episodic process of length T with histories h_{t}\in\mathcal{H} . At turn t , K agents sequentially generate proposals a_{t}^{(1:K)}=(a_{t}^{(1)},\dots,a_{t}^{(K)}) according to a joint policy \pi_{\theta} . We assume the joint proposal distribution admits the autoregressive factorization \pi_{\theta}(a_{t}^{(1:K)}\mid h_{t})=\pi_{\theta_{1}}(a_{t}^{(1)}\mid h_{t})\prod_{i=2}^{K}\pi_{\theta_{i}}\!\left(a_{t}^{(i)}\mid h_{t},a_{t}^{(1:i-1)}\right), (1) so that agent i may condition on the proposals produced by earlier agents within the same turn. This formulation captures common multi-agent pipelines such as planner–solver–critic architectures, where later agents observe earlier outputs. The frequently used independent-agent model is recovered as the special case \pi_{\theta_{i}}(a_{t}^{(i)}\mid h_{t},a_{t}^{(1:i-1)})=\pi_{\theta_{i}}(a_{t}^{(i)}\mid h_{t}), which yields the product factorization \pi_{\theta}(a_{t}^{(1:K)}\mid h_{t})=\prod_{i=1}^{K}\pi_{\theta_{i}}(a_{t}^{(i)}\mid h_{t}). (2) In the remainder of the paper, we use the autoregressive formulation 1 to model collaborative generation pipelines where agents can observe earlier proposals, and the independent factorization ( 2 ) to model competitive routing settings where agents generate proposals without access to others’ outputs. A mechanism M maps the set of proposals to the deployed output y_{t}\in\mathcal{Y} : y_{t}=M(h_{t},a_{t}^{(1:K)},\xi_{t}), where \xi_{t} captures any mechanism randomness (e.g., stochastic routing). The environment/user then transitions as h_{t+1}\sim P(\cdot\mid h_{t},y_{t}) . A scalar feedback signal is computed on the deployed output, r_{t}=r(h_{t},y_{t}) , and the return is R_{1:T}=\sum_{t=1}^{T}r_{t} . Let the reward allocation rule of the mechanism be \alpha_{i}(h_{t},a_{t}^{(1:K)},\xi_{t})\in[0,1]\text{ and }\sum_{i=1}^{K}\alpha_{i}=1, which specifies the fraction of the deployed reward assigned to agent i . The expected utility of agent i at turn t is defined as U_{i}:=\mathbb{E}\!\left[\alpha_{i}(h_{t},a_{t}^{(1:K)},\xi_{t})\,r(h_{t},y_{t})\right], where the expectation is over proposal generation and mechanism randomness. The system objective is J_{\mathrm{sys}}(\theta_{1:K}):=\mathbb{E}[\sum_{t=1}^{T}r(h_{t},y_{t})]. We keep r(\cdot) abstract here; in implementation we will instantiate r using a concrete reward construction (e.g., multiturn-aware reward). Filtered feedback as partial observability of potential outcomes For any fixed (h_{t},a_{t}^{(1:K)},\xi_{t}) , define the deployed outcome y_{t} and reward r_{t}=r(h_{t},y_{t}) . Learning does not necessarily observe all potential outcomes such as r(h_{t},y_{t}^{\prime}) for alternative outputs y_{t}^{\prime} that were not

Read the complete text