Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution
Tatiana Anikina, Arne Binder, David Harbecke, Stalin Varanasi, Leonhard Hennig, Simon Ostermann, Sebastian Möller, Josef van Genabith · 2025
Read Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution by Tatiana Anikina, Arne Binder, David Harbecke, Stalin Varanasi, Leonhard Hennig, Simon Ostermann, Sebastian Möller, Josef van Genabith (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19316.txt.
Introduction
Anikina and seven colleagues test whether simple task training helps a hard language task in twenty twenty-five. Coreference resolution links pronouns and names to the things they mean. It needs deep context understanding. Classic probing checks what a model's layers know, but usually in the other direction.
Their reverse probing trains models on simpler tasks first. Source tasks included paraphrase detection, entity recognition, and relation extraction. Embeddings from those tasks were then applied to coreference. Paraphrase training helped most, matching the need for meaning similarity. Middle layers of fine tuned models often beat final layers. Combining several task embeddings improved results further. Attention based merging gave the largest gains.
Catalog record
| CREATOR | Tatiana Anikina, Arne Binder, David Harbecke, Stalin Varanasi, Leonhard Hennig, Simon Ostermann, Sebastian Möller, Josef van Genabith |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19316
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution Tatiana Anikina, Arne Binder, David Harbecke, Stalin Varanasi, Leonhard Hennig, Simon Ostermann, Sebastian Möller, Josef van Genabith Abstract In this work, we reimagine classical probing to evaluate knowledge transfer from simple source to more complex target tasks. Instead of probing frozen representations from a complex source task on diverse simple target probing tasks (as usually done in probing), we explore the effectiveness of embeddings from multiple simple source tasks on a single target task. We select coreference resolution, a linguistically complex problem requiring contextual understanding, as focus target task, and test the usefulness of embeddings from comparably simpler tasks such as paraphrase detection, named entity recognition, and relation extraction. Through systematic experiments, we evaluate the impact of individual and combined task embeddings. Our findings reveal that task embeddings vary significantly in utility for coreference resolution, with semantic similarity tasks (e.g., paraphrase detection) proving most beneficial. Additionally, representations from intermediate layers of fine-tuned models often outperform those from final layers. Combining embeddings from multiple tasks consistently improves performance, with attention-based aggregation yielding substantial gains. These insights shed light on relationships between task-specific representations and their adaptability to complex downstream tasks, encouraging further exploration of embedding-level task transfer. Our source code is publicly available. 1 1 1 github.com/Cora4NLP/multi-task-knowledge-transfer † † * Equal contribution. Figure 1: Probing workflow with Coreference Resolution (Coref) as target task and four different source tasks: Relation Extraction (RE), Question Answering (QA), Named Entity Recognition (NER), and Paraphrase Detection (MRPC). 1 Introduction Language models have exhibited superior performance in most areas of NLP applications, including natural language inference Williams et al. (2018) , question answering Rajpurkar et al. (2016) ; Rajpurkar et al. (2018) , commonsense reasoning Talmor et al. (2019) ; Ostermann et al. (2019) , and others. Since the establishment of language models with partial super-human performance, research has aimed to pinpoint which types of knowledge are exactly encoded by such language models. One technique in the field of explainable artificial intelligence for evaluating the presence of such types of knowledge is probing Conneau et al. (2018) ; Hewitt and Liang (2019) ; Tenney et al. (2019a) ; Belinkov (2022) . Probing involves adding linear classifiers on top of representations extracted from a pre-trained model, which are trained on simple tasks for predicting a feature of choice, such as syntactic structures Lin et al. (2019) , entity types Tenney et al. (2019b) , or specific types of commonsense knowledge Zhou et al. (2020) . A main intuition behind probing is to evaluate to what degree the representations that are learned from the complex source task can be re-purposed to solve a new, simpler task Belinkov (2022) . In our work we decide to reverse this paradigm (thus reverse probing ) and investigate how different source task embeddings, from a model trained on simple tasks, can be adapted to a new, more complex target task. In other words, we try to answer the question: Can we reuse knowledge from simpler tasks for a more complex task? Such a recycling of knowledge is not only interesting to deepen our understanding of what type of knowledge is encoded in language models, but it also results in more energy-efficient deep learning, by reusing network weights and representations. We choose coreference resolution Lee et al. (2017) as our target task because solving coreference is - up to date - a challenging NLP problem that even newer large language models struggle with Bohnet et al. (2023) ; Martinelli et al. (2024) . Coreference resolution involves understanding of context, what counts as a valid mention and which mentions refer to the same entity. Solving coreference may require different types of linguistic knowledge. Our goal is to find out which types of information from which source task models are useful and how this information can be combined and/or adapted to work for the target task. To isolate the effects of single tasks, we rely on small language models, in our case BERT Devlin et al. (2019) . Such models do not possess sophisticated in-context abilities and require finetuning steps in order to perform well on tasks. Our research questions are as follows: (1) Which source tasks are beneficial for combination into a more complex target task, here coreference resolution? (2) Which layers of source task models contribute most to the target model performance? (3) What are the effects of combining embeddings from different models and layers? How should these embeddings be aggregated? Can we improve word representations by extending the embedding context and combining the outputs of several hidden layers? 2 Reverse Probing The goal of our framework is to evaluate the transferability of knowledge embedded in representations from simpler source tasks to a complex target task. Figure 1 gives an overview. Let S=\{s_{1},s_{2},\dots,s_{k}\} be a set of source tasks with models pre-trained on simpler NLP tasks, and T be the target task (coreference resolution in this case). M_{s} is a pre-trained model fine-tuned on source task s . We then define H^{l}_{s} to be the output embeddings from layer l of M_{s} . For each source task s_{i}\in S , we extract embeddings \mathbf{H}_{s} from layer l of the corresponding source task model M_{s} (Figure 1 , embedding extraction). We either take the output at a single or multiple consecutive layers. Note that these may be also intermediate layers (model truncation). Optionally, we apply L 2 normalization. Secondly, we aggregate token embeddings from different source task models by using an aggregation function A to combine embeddings across layers and models. The aggregation is done token-wise, so that every token can be represented as a combination of different model outputs. We define A to be either the mean of all vectors, i.e. as \mathbf{E}_{T}=\frac{1}{k}\sum_{i=1}^{k}\mathbf{H}_{s_{i}} Alternative, we use a simple attention mechanism for the combination (Figure 1 , embedding aggregation): \mathbf{E}_{T}=\sum_{i=1}^{k}\alpha_{i}\mathbf{H}_{s_{i}} where \alpha_{i}=\text{softmax}(\mathbf{W}\cdot\mathbf{H}_{s_{i}}) In some experiments we use only a single model. In this case the mean corresponds to the original embedding of the source model and attention simply means self-attention. Next, the aggregated token embeddings are passed to the target task head that includes several layers with trainable weights (Figure 1 , training target task layers). Figure 1 shows the probing workflow with four different source task models. Each source model is pre-trained separately on a corresponding dataset as described in Section 3 . The models are based on bert-base-cased contextualized embeddings with different task-specific heads and their weights are frozen. Given that the source models cannot update their weights during probing, our conjecture is that those models that perform better on the coreference task “out-of-the-box" contain some useful information that is relevant for the target task. 3 Tasks In this section we describe the target task, the source tasks and their respective training data. 3.1 Target Task As our target model we choose a popular end-to-end coreference resolution model based on the implementation by Xu and Choi (2020) and train it on the OntoNotes CoNLL 2012 corpus Pradhan et al. (2012) . We use bert-base-cased and the recommended parameters for fine-tuning 2 2 2 https://github.com/lxucs/coref-hoi/blob/master/experiments.conf . 3.2 Source Tasks We focus on the comparison with standard BERT as well as four other