When More Data Doesn’t Help: Limits of Adaptation in Multitask Learning

Steve Hanneke, Mingyue Xu · 2026

Read When More Data Doesn’t Help: Limits of Adaptation in Multitask Learning by Steve Hanneke, Mingyue Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20774.txt.

Introduction

Steve Hanneke and Mingyue Xu find cases where extra data from sister tasks stops helping. Their 2026 paper studies the limits of adaptation in multitask learning. Multitask learning pools datasets from related source tasks. The hope is to beat what each task could achieve alone. The framework has scored tremendous successes in modern applications. The authors ask when that hope fails. Their answer draws a boundary around the method's promise.

Adaptation means bending a shared model toward one target task. If the source tasks differ too much, pooled data can mislead. The paper characterizes when transfer gains vanish. Such limits guide practitioners who must choose between pooling and going solo. Knowing a method's failure zone is as useful as knowing its strengths. The work is learning theory with direct design advice.

Catalog record

CREATORSteve Hanneke, Mingyue Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE113 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20774

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When More Data Doesn’t Help: Limits of Adaptation in Multitask Learning Steve Hanneke, Mingyue Xu Abstract Multitask learning and related frameworks have achieved tremendous success in modern applications. In multitask learning problem, we are given a set of heterogeneous datasets collected from related source tasks and hope to enhance the performance above what we could hope to achieve by solving each of them individually. The recent work of 27 has showed that, without access to distributional information, no algorithm based on aggregating samples alone can guarantee optimal risk as long as the sample size per task is bounded. In this paper, we focus on understanding the statistical limits of multitask learning. We go beyond the no-free-lunch theorem in 27 by establishing a stronger impossibility result of adaptation that holds for arbitrarily large sample size per task. This improvement conveys an important message that the hardness of multitask learning cannot be overcame by having abundant data per task. We also discuss the notion of optimal adaptivity that may be of future interests. 1 Introduction Multitask learning and related concepts such as multi-source domain adaptation ( 13 ; 12 ) , transfer learning ( 35 ; 15 ) , and meta learning/learning-to-learn ( 40 ; 23 ; 16 ) , to name a few, have become increasingly popular in modern applications of science and engineering, especially when data availability is an issue. Possibly, data generated by multiple related sources could be aggregated towards solving multiple learning tasks more efficiently than we could hope to solve each of them on their own. Such examples in reality include drug discovery where predicting a molecule’s efficacy and safety often benefits from jointly learning across many related assays and targets, robotics multi-skill learning where training perception and control jointly for several manipulation or navigation skills can accelerate learning new tasks, etc. In this paper, we study the problem of multitask learning from a statistical perspective, where independent datasets Z_{t}\sim P_{t},t=1,2,\ldots are collected from multiple sources to be used for improving performance over learning individually. As a common belief, a single aggregation of datasets could benefit all tasks simultaneously. However, leveraging those noisy datasets might hurt the performance on target (any single such task). This motivates the notion of “adaptivity”, asking for algorithms that can automatically identify good datasets, without possessing additional distributional information. The recent work of 27 studied the hardness of multitask learning in terms of adaptivity. They first provided minimax upper and lower bounds for multitask learning rates accounting for the number of tasks, sample size per task, as well as certain notions of discrepancy between distributions. Their main result then revealed that adaptivity is impossible in general, namely, no learning procedure can achieve minimax rates as long as the sample size per task is constant and the learner has no additional information apart from the given multi-source data (see Subsection 3.1 for a detailed discussion). Our work is built on top of theirs. Specifically, we address an open question remained in their paper, that is, whether having more samples per task could perhaps yield some quantifiable advantages from having multiple sources (e.g., by comparing and ranking them). To make it clear, their negative results followed from a probabilistic construction of multitasks where sample size per task needs to be bounded. Therefore, whether adaptivity is possible when having a larger number of samples per task remains open. Our work answers negatively to this conjecture by showing a stronger impossibility result for adaptation that holds for arbitrarily large sample size per task. Notably, this improvement is substantial as it conveys an important message that the failure of adaptivity in multitask learning cannot be overcame by collecting abundant data per source task. In other words, having enough randomness in multitask distributions and sufficiently many tasks makes it hard to identify the optimal aggregation of datasets, no matter how many samples can be used for comparison. Our proof is technical, involving an application of the Fano’s method together with a tight upper bound of the KL divergence between mixture distributions. Furthermore, our results raise a fundamental problem of optimal adaptivity, i.e., what is the optimal adaptive rates (optimal rates achieved by algorithms with only access to multisource datasets) for multitask learning and what adaptive algorithms can achieve such optimal adaptive rates, which could be an interesting future research direction. 1.1 Related Works Theory of multitask learning, dating back to 11 , has been extensively explored in the past. Below, we provide a brief overview of existing theoretical works on multitask learning and related areas. 18 ; 14 ; 13 ; 12 built theoretical works on multitask learning based on distributional measures such as d_{\mathcal{A}} -divergence and the \mathcal{Y} -discrepancy. 37 investigated the optimal data aggregation in multitask learning using a notion of non-metric discrepancy called the Rényi divergence. Ideally speaking, aggregation of multitask datasets could possibly benefit all tasks simultaneously. This has motivated a line of works ( 39 ; 42 ; 40 ) aiming to provide guarantees on average risk across tasks. For the adversarial robustness of multitask learning, the works of 44 ; 43 have derived positive and negative results under various adversarial corruption in datasets. For multitask PAC learning with adversarial corruptions, 33 showed that there is a learning algorithm that overcomes the effect of corruptions and approaches the optimal test error as long as less than half of the sources are manipulated. Another bulk of theoretical works focused on the topic of multitask meta/representation learning where tasks are related by a shared latent representation/subspace ( 41 ; 6 ; 46 ; 20 ; 45 ; 22 ; 17 ; 5 ; 1 ; 50 ) . Such settings are often formalized with particular structural assumptions on the shared substructures (e.g., a low-dimensional linear mapping), which are not the focus of the present work. Apart from what has already been discussed, there are other relevant literature includes online multitask learning ( 7 ; 19 ) , federated/distributed learning ( 49 ; 51 ) , and distance-based multitask learning ( 21 ; 24 ) . 1.2 Organization The remaining of the paper is organized as follows. In Section 2 , we formally describe the problem setup for multitask learning and introduce necessary definitions. In Section 3 , we review the main results of the previous work and outline our contributions relatively. In Section 4 , we consider the special case of agnostic learning and reveal the suboptimality of the pooling (global ERM) algorithm among adaptive algorithms. In Section 5 , we present our main result which is an extended impossibility result for adaptation under a general noise assumption as well as a detailed technical overview. In Section 6 , we conclude the paper and propose some interesting future research directions. All proofs are deferred to the appendix. 2 Preliminaries We consider a binary classification setting. Let \mathcal{X} be an instance space and \mathcal{Y}=\{0,1\} . Let \mathcal{H}\subseteq\mathcal{Y}^{\mathcal{X}} be a concept class with bounded VC dimension ( \text{VC}(\mathcal{H})=d_{\mathcal{H}}<\infty ), which we consider fixed in all subsequent discussions. Let P be a data distribution on \mathcal{X}\times\mathcal{Y} , let S_{n}=\{(x_{i},y_{i})\}_{i=1}^{n}\in(\mathcal{X}\times\mathcal{Y})^{n} be a dataset. For any concept h\in\mathcal{H} , define its error rate as \text{er}_{P}(h)=P\{(x,y):h(x)\neq y\} and its empirical error

Read the complete text