CAIR : Counterfactual-based Agent Influence Ranker for Agentic AI Workflows
Amit Giloni, Chiara Picardi, Roy Betser, Shamik Bose · 2025
Read CAIR : Counterfactual-based Agent Influence Ranker for Agentic AI Workflows by Amit Giloni, Chiara Picardi, Roy Betser, Shamik Bose (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25612.txt.
Introduction
Amit Giloni led five co-authors introducing CAIR. Chiara Picardi, Roy Betser, and Shamik Bose helped. Aishvariya Priya Rathina Sabapathy and Roman Vainshtein followed. Their 2025 paper ranks agent influence in AI workflows. Agentic workflows assemble several language model agents collaboratively. They work autonomously toward a shared goal together. Understanding their operations matters for quality and security.
No existing methods assess each agent influence level. Static structural analysis is unsuitable for inference-time execution. Their ranker is the first such method. Counterfactual analysis compares output with agents removed. The analysis is task-agnostic in its design. It can run offline and at inference time. Evaluation used their own dataset of thirty use cases. CAIR produces consistent rankings across those cases. It outperforms baseline methods in their evaluation. Downstream task effectiveness can improve using its rankings. Security audits of multi-agent systems may also benefit.
Catalog record
| CREATOR | Amit Giloni, Chiara Picardi, Roy Betser, Shamik Bose |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25612
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CAIR : Counterfactual-based Agent Influence Ranker for Agentic AI Workflows Amit Giloni, Chiara Picardi, Roy Betser, Shamik Bose Abstract An Agentic AI Workflow (AAW), also known as an LLM-based multi-agent system, is an autonomous system that assembles several LLM-based agents to work collaboratively towards a shared goal. The high autonomy, widespread adoption, and growing interest in such AAWs highlight the need for a deeper understanding of their operations, from both quality and security aspects. To this day, there are no existing methods to assess the influence of each agent on the AAW’s final output. Adopting techniques from related fields is not feasible since existing methods perform only static structural analysis, which is unsuitable for inference time execution. We present Counterfactual-based Agent Influence Ranker (CAIR ) - the first method for assessing the influence level of each agent on the AAW’s output and determining which agents are the most influential. By performing counterfactual analysis, CAIR provides a task-agnostic analysis that can be used both offline and at inference time. We evaluate CAIR using an AAWs dataset of our creation, containing 30 different use cases with 230 different functionalities. Our evaluation showed that CAIR produces consistent rankings, outperforms baseline methods, and can easily enhance the effectiveness and relevancy of downstream tasks. 1 Introduction An LLM-based AI agent is a computer program that independently gathers information from its surroundings, analyzes it, makes informed decisions, and performs actions to accomplish defined goals John et al. 2025 . Such AI agents are used in various domains and tasks, including productivity in the workplace Ðula et al. 2024 , and coding assistants Pinto et al. 2024 . To tackle complex and multi-disciplinary tasks, several agents can be assembled into an Agentic AI Workflow (AAW), also known as an LLM-based multi-agent system, to work collaboratively towards a shared goal Yang et al. 2024 . The market of such AI agents and AAWs is projected to experience an eightfold increase in adoption over the next six years 1 1 1 marketsandmarkets.com/Market-Reports/ai-agents-market-15761548.html . Figure 1: Overview of CAIR phases. This growing interest, particularly in AAWs, raises the need for a deeper understanding of the AAW’s operation and the relations between its agents. Moreover, due to the high autonomy of the AAW, it is important from both security and quality perspectives to understand the influence level of each agent on the AAW’s final output. This deeper understanding will be highly beneficial when applying additional quality and trust-supported downstream tasks on the AAW. These may include observability and monitoring Dong et al. 2024 , as well as failure detection mechanisms Su et al. 2024 . An example of such a solution is AI guardrails for harmful content - LLM-level safety mechanisms that enforce the LLM to comply with ethical guidelines Kumar et al. 2023 . Applying such guardrails on every LLM call performed by any agent within the AAW can introduce significant latency, potentially tripling the system’s inference time. Theoretically, identifying the agents that have a greater influence on the AAW’s output will provide interpretability and understanding of the AAW. Selectively applying guardrails to those agents will substantially reduce latency while still preventing the generation of harmful content. There are currently no methods for assessing each agent’s influence on the AAW’s final output. Adopting solutions from related fields of study, such as graph theory Saxena and Iyengar 2020 , communication network security Mell 2021 ; Imran et al. 2013 , and reinforcement learning Chen et al. 2024 , to be used on AAW is not feasible since AAWs are dynamic by nature and those solutions perform a static analysis. These solutions: 1) would analyse only the AAW architecture and not the AAW internal operation (e.g., agent’s capabilities, final output, agents’ outputs, etc.); 2) do not support a flexible AAW that changes the activated agents according to the input query; and 3) are not timely feasible for use at AAW inference. We present CAIR - the first method for assessing the influence level of agents on the AAW’s output and determining which agents are the most influential. As illustrated in Figure 1 , CAIR has two phases - offline and online. CAIR performs a deep task- and architecture-agnostic offline analysis of the AAW and leverages its insights to perform a fast and efficient online assessment, predicting the agents’ expected influence rankings. In its offline analysis, CAIR leverages concepts from classical machine learning feature importance assessments. It begins by using a limited set of representative queries to trigger AAW activations. Then it iterates over all activated agents and injects counterfactual agents’ outputs. Finally, it assesses the impact of the counterfactual change on the final AAW output. At inference, CAIR selects the agents’ rankings based on the offline analysis results. We evaluate CAIR using AAW-Zoo - a dataset of our creation, containing 30 AAW use cases from three common architectures, comprising a total of 230 distinct functionalities. AAW-Zoo was created by our AAW-Zoo-Generator - a dedicated LLM-chain that creates AAWs for an input use case description. Our evaluation showed that CAIR : 1) substantially outperforms baselines adopted from other fields; 2) is correlated with the ground truth, both in offline and online settings; and 3) produces consistent rankings for similar queries. Moreover, we showed that using CAIR as part of a downstream task reduces the latency by 27%, with a minimal performance effect. This research’s contributions are as follows: • The first one-of-a-kind agent influence ranker for agentic AI workflows. CAIR is task- and architecture-agnostic and can be used in both offline and online analysis, with negligible added latency to the workflow inference time. • The first to enable downstream tasks designed for LLMs to be applied to multi-agent AAWs efficiently during inference. • A pioneering analysis of AAWs that provides interpretability and understanding of AAWs. • CAIR code 2 2 2 https://github.com/FujitsuResearch/CAIR.git , the AAW-Zoo dataset and the AAW-Zoo-Generator 3 3 3 https://github.com/FujitsuResearch/Auto_Agent_Creation.git used to create the dataset can be used by the research community to further investigate AAWs and their nature in a variety of quality and security aspects. 2 Related Works The high autonomy of AAWs causes them to be unpredictable and uninterpretable Watson et al. 2024 . Currently, there are no existing methods for interpreting the final output of an agentic workflow (AAW) concerning the individual agents involved, nor for assessing their respective influence on the outcome. However, solutions can be adopted from related fields. Figure 2: CAIR ’s offline phase. For each representative query, CAIR creates counterfactual agents’ outputs for each agent. The influence score is calculated based on the resulting changes via the amplification factor (AF), agent output change (AOC), final output change (FOC), complete output change (OC), and workflow change (WC). Viewing the AAW as a connectivity graph by considering each agent as a node and each connection between agents as an edge enables the use of measures from the graph theory field. In the graph theory field, the importance of each node is computed by centrality measures Saxena and Iyengar 2020 . The mostly used measures are Betweenness centrality Saxena and Iyengar 2020 , which measures how often a node appears on the shortest paths between other nodes in a network, and Eigenvector centrality Saxena and