Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors

Jingquan Yan, Yuwei Miao, Peiran Yu, Junzhou HuangComputer Science and Engineering DepartmentUniversity of Texas at Arlington {jingquan.yan,yuwei.miao,peiran.yu,jzhuang}@uta.edu · 2026

Read Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors by Jingquan Yan, Yuwei Miao, Peiran Yu, Junzhou HuangComputer Science and Engineering DepartmentUniversity of Texas at Arlington {jingquan.yan,yuwei.miao,peiran.yu,jzhuang}@uta.edu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17898.txt.

Description

Jingquan Yan and team analyze optimization and capacity limits of attention-based regressors, breaking the correlation plateau. Attention models train by optimizing error and correlation losses, weighting error size and order or shape. A common training effect is the PCC plateau: correlation stops improving early even as error keeps falling.

The paper gives a rigorous theory of that behavior, revealing limits in tuning motion and model ceiling. Lowering error can oddly suppress the correlation slope, a conflict worsened by softmax attention, above all for highly uniform data. A capacity limit appears: for any convex mixer including softmax attention, the convex hull of inputs bounds possible correlation gain. Data sameness intensifies both limits. Motivated by those insights, the authors propose Extrapolative Correlation Attention, with tricks to improve correlation tuning and reach past the convex hull. Across benchmarks including uniform settings, it breaks the plateau, lifting correlation without hurting error scores.

Catalog record

CREATORJingquan Yan, Yuwei Miao, Peiran Yu, Junzhou HuangComputer Science and Engineering DepartmentUniversity of Texas at Arlington {jingquan.yan,yuwei.miao,peiran.yu,jzhuang}@uta.edu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-17898

Browse the full library · About the Tower of Babel