Model Proficiency in Centralized Multi-Agent Systems: A Performance Study

Anna Guerra, Francesco Guidi, Pau Closas · 2025

Read Model Proficiency in Centralized Multi-Agent Systems: A Performance Study by Anna Guerra, Francesco Guidi, Pau Closas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23447.txt.

Introduction

Anna Guerra, Francesco Guidi, Pau Closas, Davide Dardari, and Petar M Djuric study how a team of autonomous agents can judge its own proficiency in a centralized multi-agent system. An agent ability to do a task depends on both individual and team skill. Proficiency self-assessment is known for single agents but largely unexplored for teams.

They present a framework for team self-assessment and investigate three metrics: the measurement prediction bound, the Kolmogorov-Smirnov statistic, and the Kullback-Leibler divergence. Each quantifies a gap between expected and observed performance. Teams that know their limits can ask for help before failing.

Catalog record

CREATORAnna Guerra, Francesco Guidi, Pau Closas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23447

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Model Proficiency in Centralized Multi-Agent Systems: A Performance Study Thanks: Anna Guerra and Davide Dardari are with the University of Bologna - DEI “Guglielmo Marconi”, Italy. E-mail: {anna.guerra3, davide.dardari}@unibo.it Francesco Guidi is with the CNR, IEIIT, Bologna, Italy. E-mail: [email protected]. Pau Closas is with ECE, Northeastern University, Boston, USA. E-mail: [email protected]. Petar M. Djurić is with ECE, Stony Brook University, Stony Brook, NY 11794, USA. E-mail: [email protected]. This work was partially supported by the European Union under the Italian National Recovery and Resilience Plan (NRRP) of NextGenerationEU, partnership (Mission 4 – Component 2 -Investment 1.1) Prin 2022 (No. 104, 2/2/2022, CUP J53C24002790006), under ERC Grant no. 101116257 (project CUE-GO: Contextual Radio Cues for Enhancing Decision Making in Networks of autonomous agents), and by the NSF under Awards 2212506, 1845833, 2326559 and 2530870. Anna Guerra, Francesco Guidi, Pau Closas Abstract Autonomous agents are increasingly deployed in dynamic environments where their ability to perform a given task depends on both individual and team-level proficiency. While PSA ( PSA ) has been studied for single agents, its extension to a team of agents remains underexplored. This letter addresses this gap by presenting a framework for team PSA in centralized settings. We investigate three metrics for centralized team PSA: the MPB ( MPB ), the KS ( KS ) statistic, and the KL ( KL ) divergence. These metrics quantify the discrepancy between predicted and actual measurements. We use the KL divergence as a reference metric since it compares the true and predictive distributions, whereas the MPB and KS provide efficient indicators for in situ assessment. Simulation results in a target tracking scenario demonstrate that both MPB and KS metrics accurately capture model mismatches, align with the KL divergence reference, and enable real-time proficiency assessment. I Introduction In recent years, AA have rapidly expanded as a means to address increasingly complex problems under real-time constraints. To fully harness their potential, there is a need to design coordinated teams of AA that can carry out demanding tasks while evaluating both their individual and team proficiency, thus engaging in ongoing learning throughout their operation [ 1 , 2 , 3 ] . PSA is not a new concept and has already been applied in multiple disciplines, such as explainable AI ( AI ) [ 4 ] , safety applications [ 5 ] , and robotics [ 6 , 7 ] . Several recent surveys provide a comprehensive overview of this evolving field [ 8 , 9 , 10 ] . Broadly speaking, PSA refers to an agent’s ability to anticipate, estimate, and evaluate its own capacity to accomplish a task [ 10 ] . This ability enables AA to refine their decision-making and interactions while operating in dynamic, time-varying environments. In this work, PSA validates the reliability of the models that agents use to interpret their surroundings [ 11 ] . PSA methodologies have traditionally been divided into three main categories, according to whether they rely on an expert (e.g., human), machine learning, or statistics [ 8 ] . This work focuses on test-based PSA performed in situ (during execution), which leverages statistical techniques [ 12 , 13 , 14 , 15 ] . In signal processing, model validation frequently relies on statistical hypothesis testing to assess regression and generalized linear models [ 11 , 16 , 17 , 18 ] , as well as to identify regime-switching models [ 19 , 20 ] . For example, [ 18 ] proposed a method for evaluating dynamic nonlinear models based on empirical and predictive cumulative distributions together with the KS statistic. Similarly, in [ 21 ] , Bayesian CRLB ( CRLB ) derived from predictive distributions was employed to quantify model proficiency. Overall, PSA is fundamental to improving efficiency, adaptability, resource allocation, fault tolerance, and system reliability. However, while significant progress has been made in equipping individual agents with PSA capabilities, advancing team-level proficiency assessment remains underexplored. This letter presents an analysis of team proficiency in centralized network architectures, combining statistical and information-theoretic metrics. Specifically, we examine the application of the MPB , the KS statistic, and the KL divergence to a team of AA . Their effectiveness is evaluated in a moving-target tracking scenario, demonstrating that overall tracking performance can be enhanced by team proficiency, which identifies agents that rely on inaccurate observation models. II Research Problem Let us consider a network of N_{\text{A}} AA , where the agents observe a hidden (unknown) state, denoted with the RV ( RV ) \boldsymbol{\mathsf{x}}_{t} at time instant t . Each agent is assumed to collect measurements of the environment’s state, which are then shared across the network. The joint observation vector is given by \boldsymbol{\mathsf{y}}_{t}=\left[\boldsymbol{\mathsf{y}}_{1,t}^{T},\,\cdots,\,\boldsymbol{\mathsf{y}}_{i,t}^{T},\,\cdots,\,\boldsymbol{\mathsf{y}}_{N_{\text{A}},t}^{T}\right]^{T}, with \boldsymbol{\mathsf{y}}_{i,t} being the observations collected by the i -th sensors. By processing the information gathered by the agents, the network aims to estimate the hidden state \hat{\boldsymbol{\mathsf{x}}}_{t} . Formally, this estimate is given by the conditional expectation of the state \hat{\boldsymbol{\mathsf{x}}}_{t}=\mathbb{E}\left[\boldsymbol{\mathsf{x}}_{t}\lvert\boldsymbol{\mathsf{y}}_{1:t-1};\mathcal{M}\right] , where the expectation is taken in accordance with the SSM ( SSM ) defined as \displaystyle\mathcal{M} \displaystyle\triangleq\left\{\mathcal{M}_{x},\,\mathcal{M}_{y}\right\} \displaystyle=\left\{f_{\boldsymbol{\mathsf{x}}_{t}\lvert\boldsymbol{\mathsf{x}}_{t-1}}(\boldsymbol{x}_{t}\lvert\boldsymbol{x}_{t-1},\,\boldsymbol{\theta}_{x}),\,f_{\boldsymbol{\mathsf{y}}_{i,t}\lvert\boldsymbol{\mathsf{x}}_{t}}(\boldsymbol{y}_{i,t}\lvert\boldsymbol{x}_{t},\,\boldsymbol{\theta}_{y})\right\}\,, (1) with \mathcal{M}_{x} , \mathcal{M}_{y} being models of the state and observation dynamics, respectively, \boldsymbol{\theta}=\left\{\boldsymbol{\theta}_{x},\,\boldsymbol{\theta}_{y}\right\} being a vector of parameters, f_{\boldsymbol{\mathsf{x}}_{t}\lvert\boldsymbol{\mathsf{x}}_{t-1}}(\cdot) and f_{\boldsymbol{\mathsf{y}}_{i,t}\lvert\boldsymbol{\mathsf{x}}_{t}}(\cdot) being PDF that describe the state evolution and observations. The standard way to assess estimation accuracy is by comparing the estimate \hat{\boldsymbol{\mathsf{x}}}_{t} with the ground truth state \boldsymbol{\mathsf{x}}_{t} , which is usually not available. Consequently, performance is often assessed indirectly through the model’s ability to predict future measurements \boldsymbol{\mathsf{y}}_{t} from past observations \boldsymbol{\mathsf{y}}_{1:t-1} . The predicted observation at time t is \hat{\boldsymbol{\mathsf{y}}}_{t}=g_{\mathcal{M}}(\boldsymbol{\mathsf{y}}_{1:t-1})=\mathbb{E}\left[\boldsymbol{\mathsf{y}}_{t}\lvert\boldsymbol{\mathsf{y}}_{1:t-1};\mathcal{M}\right] . To this end, each agent uses a set of N_{\text{M}} statistical models stored in a library denoted with \mathcal{L}=\left\{\mathcal{M}_{0},\mathcal{M}_{1},\,\mathcal{M}_{2},\,\ldots,\,\mathcal{M}_{m},\ldots,\,\mathcal{M}_{N_{\text{M}}-1}\right\} with the true generative model, \mathcal{M}_{0} , being included in \mathcal{L} [ 11 ] . This paper addresses the fundamental question: how can a centralized team of AA assess its team proficiency in predicting observations using a metric that quantifies the discrepancy between predicted and actual measurements without relying on knowledge of the true hidden state, and that guides the selection of the most appropriate model? III Proficiency Analysis To address the previous question, we start by introducing three different metrics for proficiency assessment. We first discuss the MPB , originally proposed in [ 21 ] for a single agent. This metric is inspired by the Bayesian CRLB and represents a fundamental limit for the measurement error when predicting a new observation under a given model. Then, we recall the KS statistic, which compares the ECDF ( ECDF ) of the predictive distribution with that of the actual measurement distribution [ 16 ] . Unlike the previous approach, which yields a score, proficiency here is assessed through a hypothesis test (significance testing). Finally, we consider the KL divergence that directly compares the PDF of the prediction with the true PDF of the measurement. Measurement Prediction Bound (MPB) Following the definition of PSA presented in [ 21 ] for scalar observations, we define the in situ MPB as in [ 21 , Eq. 6] \displaystyle\!\!\!\mathcal{P}_{t}\left(\mathcal{M}|y_{1:t-1}\right) \displaystyle=-\mathbb{E}\left[\frac{\partial^{2}\,\ln f_{\mathsf{y}_{t}|\mathsf{y}_{1:t-1}}\left(y_{t}|y_{1:t-1};\,\mathcal{M}\right)}{\partial\,y_{t}^{2}}\right], (2) where the expectation \mathbb{E}\left[\cdot\right] is taken with respect to the predictive density f_{\mathsf{y}_{t}\,\mid\mathsf{y}_{1:t-1}}\left(y_{t}\mid y_{1:t-1};\mathcal{M}\right) . Under standard regularity conditions (differentiability, integrability, vanishing boundary terms) and for any predictor \hat{\mathsf{y}}_{t} with

Read the complete text