Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimization
Yujie Gu, Richeng Jin, Member, IEEE, Zhaoyang Zhang, Senior Member, IEEE, Huaiyu Dai, Fellow, IEEE · 2026
Read Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimization by Yujie Gu, Richeng Jin, Member, IEEE, Zhaoyang Zhang, Senior Member, IEEE, Huaiyu Dai, Fellow, IEEE (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-11584.txt.
Description
Shrinking gradients to cut communication is standard practice in federated learning, where models train across many devices, and it is usually assumed to cost almost nothing. Yujie Gu and three co-authors show the price is real: compression sharpens the loss landscape, especially when device data differ widely, which can hurt how well the trained model generalizes. A known fix, sharpness-aware minimization, seeks flat minima but misjudges the needed correction when data are heterogeneous.
Their remedy, FedSynSAM, builds synthetic data from the global model trajectory to estimate that correction more accurately, even under compression. The paper establishes convergence for the new algorithm and backs it with extensive experiments. The message is pointed: in federated learning, cheaper messages can quietly make models worse, unless the training method accounts for the damage.
Catalog record
| CREATOR | Yujie Gu, Richeng Jin, Member, IEEE, Zhaoyang Zhang, Senior Member, IEEE, Huaiyu Dai, Fellow, IEEE |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-11584