Federated Learning with Energy-Based Structured Probabilistic Inference
Dario Fenoglio, Daniil Kirilenko, Martin Gjoreski, Marc Langheinrich · 2026
Read Federated Learning with Energy-Based Structured Probabilistic Inference by Dario Fenoglio, Daniil Kirilenko, Martin Gjoreski, Marc Langheinrich (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30161.txt.
Introduction
Dario Fenoglio, Daniil Kirilenko, Martin Gjoreski, and Marc Langheinrich refine how federated learning servers combine client updates. Federated learning trains one model across many data owners. Servers usually average client models with fixed or heuristic weights. When client data differ wildly, those rules underperform badly. The authors let a conditional random field set the weights.
The field assigns unary potentials to single clients. Pairwise terms capture links between every pair of clients. Client reliability and client interplay are modeled together this way. Inference in the field produces the aggregation weights. Those weights aim at better convergence of the global objective. Experiments focus on non-IID data, where client datasets differ most. The method consistently beats well-established standard baselines there. Structured inference, they argue, suits server side combining well.
Catalog record
| CREATOR | Dario Fenoglio, Daniil Kirilenko, Martin Gjoreski, Marc Langheinrich |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30161
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Federated Learning with Energy-Based Structured Probabilistic Inference Dario Fenoglio, Daniil Kirilenko, Martin Gjoreski, Marc Langheinrich Abstract Federated learning typically aggregates client updates using fixed or heuristic weighting rules, which can be suboptimal when clients have heterogeneous data and varying contributions to the global model. We propose a framework that refines client aggregation weights using Conditional Random Fields (CRFs). Our method defines unary potentials for individual clients and pairwise potentials for all client pairs, allowing the server to model both client-specific reliability and interactions between clients. The resulting CRF inference produces aggregation weights that enable better convergence of the global training objective. Experiments show that, under non-IID heterogeneity, our approach consistently improves performance over well-established federated learning baselines. 1 Introduction Federated learning (FL) trains a shared model from decentralized data under the coordination of a server, while keeping each client’s raw examples local ( 15 ; 7 ) . This setting is important when data are privacy-sensitive, legally constrained, too large to centralize, or naturally generated at the edge. Practical deployments have shown that FL can support large-scale model training on mobile and cross-device populations ( 2 ) , and the same abstraction also applies to cross-silo settings such as hospitals, banks, and organizations that cannot pool data directly. The standard optimization approach in such cases is Federated Averaging (FedAvg), where clients run several local SGD steps and the server averages their returned models with weights proportional to local sample counts ( 15 ) . Its simplicity is also its limitation: once clients have heterogeneous label distributions, feature distributions, data quality, or local optimization trajectories, a sample-count weighted average treats all returned directions as equally reliable after scaling ( 5 ; 13 ) . This can lead to client drift and slow or unstable convergence under non-IID data ( 14 ; 8 ) . Existing responses include modifying the local objective with a proximal term ( 14 ) , correcting client drift with control variates ( 8 ) , normalizing heterogeneous local progress ( 18 ) , or replacing averaging with robust aggregation rules such as Krum, coordinate-wise median, trimmed mean, and geometric median aggregation ( 1 ; 19 ; 16 ) . These methods improve important failure modes, but most server-side aggregation rules remain fixed heuristics that do not explicitly model both client reliability and client-client interactions in a single inference problem. We propose to view federated server aggregation as structured probabilistic inference. Conditional Random Fields (CRFs) define an energy over a collection of dependent random variables conditioned on observations ( 11 ) . In our setting, the graph nodes are participating clients in a communication round, the observed variables are their local model deltas, and the hidden variables are discrete client reliability labels. Unary potentials score each client against a robust reference update, while pairwise potentials connect clients with similar updates and encourage them to receive compatible reliability labels. Approximate CRF inference then produces posterior reliability estimates, which refine the FedAvg aggregation weights before the global model is updated. Our contributions are: • We formulate FL server aggregation as an energy-based structured inference problem over client reliability variables. • We instantiate a fully connected client-level CRF with unary potentials derived from robust update statistics and pairwise potentials derived from update affinity. • We propose a mean-field aggregation algorithm that is compatible with the standard FedAvg training loop and evaluate it against common FL and robust aggregation baselines. 2 Background 2.1 Federated Learning Consider K clients. Client k owns a local dataset \mathcal{D}_{k} with n_{k} examples, and the total number of examples is n=\sum_{k=1}^{K}n_{k} . In the standard empirical FL formulation, the goal is to learn a single model parameter vector w\in\mathbb{R}^{d} by minimizing the sample-weighted objective \min_{w}F(w)=\sum_{k=1}^{K}p_{k}F_{k}(w),\qquad p_{k}=\frac{n_{k}}{n}, (1) where F_{k}(w)=\frac{1}{n_{k}}\sum_{(x,y)\in\mathcal{D}_{k}}\ell(w;x,y) (2) is the empirical risk on client k and \ell is the task loss. More general client-weighted objectives are possible, but sample-size weighting is the canonical choice underlying FedAvg. FedAvg solves this problem through alternating local optimization and server averaging ( 15 ) . At communication round t , the server broadcasts the current global model w_{t} to a subset of participating clients S_{t} . Each client initializes w_{t,0}^{k}=w_{t} and performs E local stochastic-gradient steps, w_{t,e+1}^{k}=w_{t,e}^{k}-\eta g_{k}(w_{t,e}^{k};\xi_{t,e}^{k}),\qquad e=0,\ldots,E-1, (3) where \xi_{t,e}^{k} is a minibatch sampled from \mathcal{D}_{k} . After local training, client k sends w_{t,E}^{k} to the server, and the server computes w_{t+1}=\sum_{k\in S_{t}}\alpha_{k}^{\mathrm{FA}}w_{t,E}^{k},\qquad\alpha_{k}^{\mathrm{FA}}=\frac{n_{k}}{\sum_{j\in S_{t}}n_{j}}. (4) Equivalently, FedAvg averages client deltas w_{t,E}^{k}-w_{t} and applies the averaged delta to the global model. Multiple local steps reduce communication, but under heterogeneous client distributions they also make local trajectories drift toward different local objectives ( 14 ; 8 ) . FedProx addresses this by adding a proximal penalty around w_{t} to the local objective ( 14 ) ; SCAFFOLD uses control variates to correct client drift ( 8 ) ; and FedNova normalizes client updates when clients perform different amounts of local work ( 18 ) . Orthogonal robust aggregation methods replace the weighted mean with robust estimators such as coordinate-wise median, trimmed mean, or geometric median to reduce sensitivity to outlying or corrupted client updates ( 19 ; 16 ) . 2.2 Conditional Random Fields Conditional Random Fields (CRFs) are probabilistic graph models that define a conditional distribution over structured data ( 11 ) . Let G=(V,E) be a graph with |V|=N nodes connected by edges E . For each node i\in V , we observe an input variable x_{i}\in\mathcal{X} and associate a random variable y_{i}\in\mathcal{Y} . A CRF model defines p(\boldsymbol{y}\mid\boldsymbol{x}) with a Gibbs distribution: p(\boldsymbol{y}\mid\boldsymbol{x})\propto\exp\!\big(-E(\boldsymbol{y}\mid\boldsymbol{x})\big), (5) where \boldsymbol{x}=\{x_{i}\}_{i=1}^{N} , \boldsymbol{y}=\{y_{i}\}_{i=1}^{N} , and E(\boldsymbol{y}\mid\boldsymbol{x}) is the energy function of the configuration. For a fully-connected graph, the energy has the common form of E(\boldsymbol{y}\mid\boldsymbol{x})=\sum_{i}\psi_{u}(y_{i},x_{i})\;+\;\sum_{i<j}\psi_{p}(y_{i},y_{j},x_{i},x_{j}), (6) where \psi_{u}(y_{i},x_{i}) is the unary potential, which can be interpreted as inverse likelihood of assigning y_{i} to node i given observation x_{i} , and \psi_{p}(y_{i},y_{j},x_{i},x_{j}) is the pairwise potential encoding interactions between nodes i and j , which is typically parameterized as \psi_{p}(y_{i},y_{j},x_{i},x_{j})=\mu(y_{i},y_{j})\,\sum_{m}w_{m}k_{m}(x_{i},x_{j}), (7) where k^{(m)}(x_{i},x_{j})\geq 0 are observation-dependent affinities between nodes i and j , and \mu(y_{i},y_{j}) is a label compatibility function specifying the cost of assigning the label pair (y_{i},y_{j}) . The true Maximum a Posteriori (MAP) solution \boldsymbol{y}^{*}=\arg\max_{\boldsymbol{y}}p(\boldsymbol{y}\mid\boldsymbol{x})=\arg\min_{\boldsymbol{y}}E(\boldsymbol{y}\mid\boldsymbol{x}) (8) is practically intractable for dense graphs. A common approximate solution is the mean field approximation , which has been widely used for fully connected CRFs ( 17 ; 9 ) . Mean field introduces q(\boldsymbol{y}\mid\boldsymbol{x})=\prod_{i}q_{i}(y_{i}\mid\boldsymbol{x}) as node-factorized approximation of p(\boldsymbol{y}\mid\boldsymbol{x}) , which is iteratively updated for each node i independently given fixed q_{j} for other nodes j\neq i via: \displaystyle q_{i}(y)\propto\exp\!\bigg( \displaystyle-\psi_{u}(y;x_{i}) (9) \displaystyle-\sum_{j\neq i}\sum_{\ell\in\mathcal{Y}}q_{j}(\ell)\,\psi_{p}(y,\ell;x_{i},x_{j})\bigg). 3 Method We keep the client-side training loop unchanged and replace only the server aggregation rule. At each communication round t , each participating client i\in S_{t} returns a locally trained model w_{i,t+1} . The server computes a flattened update vector u_{i}^{t}=\mathrm{vec}(w_{i,t+1}-w_{t}), (10) using the floating-point tensors of the model state. These vectors are used only for CRF scoring and inference; the final server update remains a weighted average of the original client model states. Client graph. For each round, we define a fully connected graph G_{t}=(V_{t},E_{t}) with one node for each participating client. The observation at node i is its update u_{i}^{t} and sample count n_{i} .