Task2Vec Readiness: Diagnostics for Federated Learning from Pre-Training Embeddings

Cristiano Augusto Dias Mafuz · 2026

Read Task2Vec Readiness: Diagnostics for Federated Learning from Pre-Training Embeddings by Cristiano Augusto Dias Mafuz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10849.txt.

Introduction

Mafuz and Silva offer Task2vec readiness scores, checks that foretell federated learning results from pre-training embeddings. Federated results swing hard with client mix, yet teams lack sound ways to foresee how a federation will act. Their no-label scores, cohesion, dispersion, and density, drawn straight from client embeddings, are tried on CIFAR-10, FEMNIST, PathMNIST, and BloodMNIST with 10 to 20 clients under varied Dirichlet mix and FedAVG pooling. Readiness and final results often track above 0.9, backing the scores as a sound proxy for outcomes and a guide for client picks.

Catalog record

CREATORCristiano Augusto Dias Mafuz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29582
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10849

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Task2Vec Readiness: Diagnostics for Federated Learning from Pre-Training Embeddings Cristiano Augusto Dias Mafuz Abstract Federated learning (FL) performance is highly sensitive to heterogeneity across clients, yet practitioners lack reliable methods to anticipate how a federation will behave before training. We propose readiness indices, derived from Task2Vec embeddings, that quantifies the alignment of a federation prior to training and correlates with its eventual performance. Our approach computes unsupervised metrics—such as cohesion, dispersion, and density—directly from client embeddings. We evaluate these indices across diverse datasets (CIFAR-10, FEMNIST, PathMNIST, BloodMNIST) and client counts (10–20), under Dirichlet heterogeneity levels spanning α ∈ { 0.05 , … , 5.0 } \alpha\in\{0.05,\dots,5.0\} and FedAVG aggregation strategy. Correlation analyses show consistent and significant Pearson and Spearman coefficients between some of the Task2Vec-based readiness and final performance, with values often exceeding 0.9 across dataset × \times client configurations, validating this approach as a robust proxy for FL outcomes. These findings establish Task2Vec-based readiness as a principled, pre-training diagnostic for FL that may offer both predictive insight and actionable guidance for client selection in heterogeneous federations. 1 Introduction Federated Learning (FL) ( 7 ) has emerged as a central paradigm for collaborative model training under privacy and communication constraints ( 15 ) . However, its performance may be hindered by client heterogeneity, data imbalance, and federation size ( 11 ) . Despite significant algorithmic advances in optimization and aggregation 14 ; 8 ; 12 , practitioners still lack principled tools to predict, before training, whether a given federation is likely to succeed. This absence of pre-training diagnostics forces costly trial-and-error experimentation and slows both research progress and deployment. We introduce Task2Vec Readiness , a framework that leverages Task2Vec embeddings ( 1 ) to derive a quantitative readiness index for federated learning. Our approach transforms each client’s data distribution into a fixed-dimensional embedding via Fisher Information, and then evaluates unsupervised metrics of federation structure. In particular, we measure cohesion (average cosine similarity among client embeddings), dispersion (average distance from the federation centroid), and density (RBF-kernel similarity over pairwise Euclidean distances). These metrics can be computed before training, and together they form a readiness profile that anticipates how well the federation can support collaborative optimization. The key novelty lies in repurposing task embeddings from transfer learning into a diagnostic signal for distributed training under heterogeneity. We conduct extensive experiments across four benchmark datasets (CIFAR-10 ( 6 ) , FEMNIST ( 2 ) , PathMNIST ( 13 ) , BloodMNIST ( 13 ) ), with client counts ranging from 10 to 20 and Dirichlet non-IID partitions spanning \alpha\in\{0.05,\dots,5.0\} . Correlation analyses consistently reveal significant Pearson and Spearman coefficients between readiness metrics and final model performance, frequently exceeding 0.9 across dataset \times client conditions. This validates readiness as a robust proxy for FL performance. Importantly, the results hold across different sources of heterogeneity, indicating that readiness captures structural properties of federations rather than dataset-specific artifacts. Our contributions are twofold: (i) we introduce the first readiness index for federated learning based on task embeddings and (ii) we demonstrate its predictive validity across diverse datasets and heterogeneity regimes. By moving the focus from post-hoc evaluation to pre-training diagnostics, our framework not only provides a new lens for understanding federated learning dynamics, but also offers a practical tool for improving performance, guiding client selection, and enhancing the efficiency and reliability of federated training at scale. 2 Background 2.1 Federated Learning Federated Learning (FL) is a machine learning paradigm in which a global model is trained collaboratively by multiple entities (which may represent individual devices or entire organizations) without sharing their private data ( 3 ) . Unlike traditional centralized approaches, where all data is aggregated into a single infrastructure, FL enables training directly on decentralized datasets. This makes it especially valuable in highly regulated domains, such as healthcare and finance, where data protection laws restrict data sharing. The prototypical FL setting consists of a central server S and a set of distributed clients C , such that |C|=K , that jointly cooperate to solve a standard supervised learning task. Each client c\in C has access to its own private training set D_{c}=\{x_{c,i},y_{c,i}\}_{i=1}^{n_{c}} where, x_{c,i} denotes the i -th training sample of the c -th client, and y_{c,i} denotes its corresponding label. The goal of FL is to train a global predictive model whose architecture and parameters w\in\mathbb{R}_{d} are shared amongst all the clients and found to minimize min_{w}\sum_{c=1}^{K}p_{c}L_{c}(w;D_{c}) , where L_{c} is the local objective and p_{c}\geq 0 specifies the individual contribution of the client c such that \sum_{c=1}^{K}p_{c}=1 . Two possible configurations for p_{c} are p_{c}=\frac{1}{K} or p_{c}=\frac{n_{c}}{N} , where N=\sum_{c=1}^{K}n_{c} . The local objective function L_{c} usually is defined as the empirical risk calculated over the training set D_{c} sampled from the client’s local data distribution L_{c}(w;D_{c})=\frac{1}{n_{c}}\sum_{i=1}^{n_{c}}l(w;(x_{c,i},y_{c,i})) , where l is an instance-level loss (e.g., cross-entropy loss or squared error in the case of classification or regression tasks, respectively). In its standard form, Federated Learning employs Federated Averaging (FedAvg) ( 7 ) , which operates over iterative rounds: 1. The server S selects a subset of clients C^{(t)}\subseteq C and broadcasts the current global model w^{(t)} . 2. Each selected client updates w^{(t)} locally on its data via stochastic gradient descent for multiple epochs, producing \theta_{c}^{(t)} . 3. The server aggregates updates using an aggregation function, typically a weighted average: w^{(t+1)}=\sum_{c\in C^{(t)}}\tfrac{n_{c}}{N}\,w_{c}^{(t)}. (1) While FedAvg provides a simple and widely adopted baseline, its effectiveness in practice depends heavily on the properties of the participating clients. In particular, the global model’s convergence and performance can be significantly affected by the statistical heterogeneity of client data, varying amounts of local data, and the total number of clients involved ( 11 ) . These factors can lead to slower convergence rates or even instability during training, despite advances in robust aggregation methods ( 14 ; 8 ; 12 ) . 2.2 Task2Vec Task2Vec is a method that was initially proposed “to provide vectorial representations of visual classification tasks which can be used to reason about the nature of those tasks and their relations” ( 1 ) . In practice, in Task2Vec, a task is represented by a labeled dataset D=\{(x_{i},y_{i})\}_{i=1}^{N} . Then, the samples are passed through a pre-trained convolutional network, referred to as the probe network, and the representation of the task is captured by computing the diagonal of the Fisher Information Matrix (FIM) associated with its filter parameters. Given a model with parameters w and output distribution p_{w}(y\mid x) , the FIM is the expected covariance of the scores (gradients of the log-likelihood) with respect to the model parameters ( 1 ) . It can be defined as: F=\mathbb{E}_{x,y\sim\hat{p}(x,y)}\Big[\nabla_{w}\log p_{w}(y\mid x)\,\nabla_{w}\log p_{w}(y\mid x)^{\top}\Big], (2) where \hat{p}(x,y) denotes the empirical distribution of the dataset. Since the full FIM may be prohibitively large depending on the probe network, 1 introduce two approximations: (i) only the diagonal entries are retained, that is, it assumes that correlations between different filters in the probe network are not important, and (ii) the Fisher information is averaged across all weights within the same filter, since these weights are usually not independent. The resulting representation of the task therefore has fixed dimensionality, equal to the number of filters in the probe network.The extraction

Read the complete text