Statistical Deficiency for Task Inclusion Estimation
Loïc Fosse, Frédéric Béchet, Benoît Favre, Géraldine Damnati, Gwénolé Lecorvé, Maxime Darrin, Philippe Formont, Pablo Piantanida · 2025
Read Statistical Deficiency for Task Inclusion Estimation by Loïc Fosse, Frédéric Béchet, Benoît Favre, Géraldine Damnati, Gwénolé Lecorvé, Maxime Darrin, Philippe Formont, Pablo Piantanida (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05491.txt.
Introduction
Machine learning talks about tasks constantly, yet the field lacks a rigorous way to say when one task contains another. Loïc Fosse and co-authors build that ruler in a 2025 paper. They define task inclusion through statistical deficiency, a classic notion comparing how much information two experiments carry about an unknown state.
Because deficiency is hard to compute directly, they propose a tractable stand-in based on information sufficiency. The proxy behaves correctly on synthetic data where the true inclusion is known. Applied to real language tasks, it rebuilds the classic natural language processing pipeline from data alone. The authors offer the framework as groundwork for studying the structure of the task space itself.
Catalog record
| CREATOR | Loïc Fosse, Frédéric Béchet, Benoît Favre, Géraldine Damnati, Gwénolé Lecorvé, Maxime Darrin, Philippe Formont, Pablo Piantanida |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05491
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Statistical Deficiency for Task Inclusion Estimation Loïc Fosse, Frédéric Béchet, Benoît Favre, Géraldine Damnati, Gwénolé Lecorvé, Maxime Darrin, Philippe Formont, Pablo Piantanida Abstract Tasks are central in machine learning, as they are the most natural objects to assess the capabilities of current models. The trend is to build general models able to address any task. Even though transfer learning and multitask learning try to leverage the underlying task space, no well-founded tools are available to study its structure. This study proposes a theoretically grounded setup to define the notion of task and to compute the inclusion between two tasks from a statistical deficiency point of view. We propose a tractable proxy as information sufficiency to estimate the degree of inclusion between tasks, show its soundness on synthetic data, and use it to reconstruct empirically the classic NLP pipeline. 1 1 1 Code and data are available here 1 Introduction Having a well-defined set of tasks with known or assumed dependency relationships has historically been a key element in building or evaluating Natural Language Processing (NLP) systems. For instance, Named Entity Recognition (NER) and summarization are two well-established tasks for which annotated datasets exist, and it is commonly accepted that the summarization task, at least in the news domain, requires NER skills to be performed effectively. As a consequence, studying generated summaries from the perspective of retained named entities is a relevant evaluation angle Pagnoni et al. (2021) ; Berezin and Batura (2022) ; Akani et al. (2023) . According to this principle, a more general hypothesis is that multi-task training Caruana (1997) provides cross-task generalization Mishra et al. (2022) ; Ye (2024) ; Wang et al. (2024) ; Baxter (2000) ; Wu et al. (2020) since tasks share dependencies. However, with the rise of instruct-tuned models Wei et al. (2021) , tasks can be directly defined by prompting large language models. This dramatically increase of the addressable tasks’ space makes the notions of ground truth and labeled dataset more fuzzy, and raises many questions about the capabilities of these seemingly all-powerful models. Notably, it is unclear how many tasks a single model can correctly handle, how many parameters are necessary to capture a given task, and what proportion of the model is dedicated to language understanding, task solving, and memorization. In this paper, to advance the understanding of the notion of task and its manipulation within language models, we are interested in studying intrinsic relationships between tasks. Building on the intuition that some tasks are necessary conditions to others ( e.g. NER is a necessary condition for summarization), we propose a framework to discover statistical task inclusion relationships . Potential application to the discovery of such relations is to build smaller, more compute-efficient datasets Zamir et al. (2018) and, more generally, to build better data-mix when training models Ye et al. (2024) , or more orthogonal benchmarks. Our main approach will rely on statistical simulation methods Le Cam (1964) ; Le Cam (1996) to decide whether one task can be transformed into another. While we theoretically show the shortcomings of naively measuring cross-task performance by directly applying each model to each other task, the contributions of the paper are: • A theoretical framework for task definition and inclusion. Based on information concepts and theory, we propose a clear definition of a task and candidate notions of inclusion (independent of the notion of model). • An experimental setup for task comparison. We propose a tractable proxy to measure task inclusion through statistical reductions. • An empirical rediscovery of the NLP pipeline Experiments suggest that our framework reconstructs the expected partial order for a sample of linguistic tasks from the classical NLP pipeline. 2 Related work Discovering the task space underlying structure Turing (1950) ; Winograd (1987) is a common problem in machine learning (ML), to compare human and machine representations of tasks but more generally to leverage potential structure for more efficient learning Li and Hiratani (2025) . We list the main families of task comparison methods and discuss how our new one differs from them. Task similarity. Evaluating similarity between tasks is one of the main area of interest, as it is the crux to discovering transfer learning or meta-learning opportunities Schmidhuber (1987) ; Zhou et al. (2021) . It can be performed by comparing the same model trained on various tasks Achille et al. (2019) ; Shui et al. (2019) , by comparing the data distributions of different tasks Ethayarajh et al. (2022) . Unlike work on task similarity, we aim at finding a non-symmetrical notion of task inclusion. We discuss in App. H limitations of similarity based approaches and why the proposed setup is more suited for task relationship study. Task merging. Inspired from ensemble methods Dietterich (2000) , model merging focuses on combining several existing models to create a new one. Recent methods either use arithmetic operations Ilharco et al. (2023) ; Tao et al. (2024) ; Ortiz-Jimenez et al. (2024) ; Zhou et al. (2024) ; Zeng et al. (2025) or more complex aggregation methods Yadav et al. (2023) ; Jin et al. (2023) ; Yang et al. (2023) , with the goal of solving conflicts or interferences between models Yu et al. (2020) ; Sener and Koltun (2018) and thus tasks. While task and model merging focuses on the parameter space (whose dimension is excessively large), the tools developed here focus on the activation space. However, we propose in App. H an analysis of parameter space based approaches and we show some limitations, despite some interesting behaviors. Task Transfer. Transfer learning Torrey and Shavlik (2010) ; Hanneke and Kpotufe (2024) ; Lange et al. (2021) consists in leveraging a model pre-trained for a new task for a given task, either as an initialization point for further training or to generate useful representations. Although most of the time one uses a generic pre-trained model and trains directly for the new task, Vu et al. (2020) showed that some tasks might benefit from training on an intermediate task, effectively building a path of (easily) transferable tasks. In computer vision, this phenomenon has been studied and quantified by Bao et al. (2019) . Zamir et al. (2018) , obtained similar results showing connections between various visual tasks and were able to leverage these structures to optimize training of multitask models Zhang and Yang (2021) . Knowledge transferability between different tasks is also at the heart of modern ML and generalization as exhibited by models such as T5 Wei et al. (2021) ; Khashabi et al. (2020) , or more recently instruct-type models Zhang et al. (2023b) with multi-task training leading to significantly stronger results. Task transfer is mainly based on successive fine-tuning processes, as well as on the study of the fine-tuned models parameters geometry (Fisher information). The tools developed here focus on fine-tuned model activations enabling us to avoid certain learning costs and connect with powerful theoretical results. Probing. Understanding what is encoded in a model has been a question of interest which led to probing methods. It consists in assessing the activations of a model on a downstream task Guillaume and Yoshua (2017) ; Chen et al. (2021) ; Rogers et al. (2021) ; Wallat et