Transfer Learning through Enhanced Sufficient Representation: Enriching Source Domain Knowledge with Target Data

Yeheng Ge, Xueyu Zhou, Jian Huang · 2025

Read Transfer Learning through Enhanced Sufficient Representation: Enriching Source Domain Knowledge with Target Data by Yeheng Ge, Xueyu Zhou, Jian Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20414.txt.

Introduction

Transfer learning borrows knowledge from a data-rich source task to help a data-poor target task. Yeheng Ge, Xueyu Zhou, and Jian Huang propose TESR, a method built on sufficient representations: compact summaries that keep everything the source data knows about the answer. Their twist is to enrich that summary with an extra component learned from target data.

Because the method works through representations rather than model shapes, the source and target can use unlike model types, such as regression at the source and classification at the target. The authors prove properties of the approach and test it on simulations and real datasets. Results show solid gains with small target samples. They present TESR as a flexible route for cross-task transfer.

Catalog record

CREATORYeheng Ge, Xueyu Zhou, Jian Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-20414

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transfer Learning through Enhanced Sufficient Representation: Enriching Source Domain Knowledge with Target Data Yeheng Ge, Xueyu Zhou, Jian Huang Abstract Transfer learning is an important approach for addressing the challenges posed by limited data availability in various applications. It accomplishes this by transferring knowledge from well-established source domains to a less familiar target domain. However, traditional transfer learning methods often face difficulties due to rigid model assumptions and the need for a high degree of similarity between source and target domain models. In this paper, we introduce a novel method for transfer learning called Transfer learning through Enhanced Sufficient Representation (TESR). Our approach begins by estimating a sufficient and invariant representation from the source domains. This representation is then enhanced with an independent component derived from the target data, ensuring that it is sufficient for the target domain and adaptable to its specific characteristics. A notable advantage of TESR is that it does not rely on assuming similar model structures across different tasks. For example, the source domain models can be regression models, while the target domain task can be classification. This flexibility makes TESR applicable to a wide range of supervised learning problems. We explore the theoretical properties of TESR and validate its performance through simulation studies and real-world data applications, demonstrating its effectiveness in finite sample settings. Keywords: Conditional independence; Domain heterogeneity; Invariant representation; Representation enhancement; Supervised learning. 1 Introduction Transfer learning has emerged as a powerful paradigm in statistics and machine learning, enabling models to leverage knowledge from related tasks to enhance performance on a target task with limited data ( Pan and Yang 2009 ; Torrey and Shavlik 2010 ) . Traditional machine learning approaches often require large amounts of data, which can be costly and time-consuming to obtain. Transfer learning addresses the data scarcity problem by transferring knowledge from source domains, where data is abundant, to a target domain, where data is scarce ( Weiss et al. 2016 ) . Despite its success, existing transfer learning methods often face limitations in terms of scalability and adaptability to diverse domains ( Zhuang et al. 2020 ) . In this paper, we propose a novel transfer learning method that overcomes these limitations by introducing a more flexible framework. This framework is capable of efficiently adapting to various target domains while maintaining high performance. Our approach builds upon recent advancements in sufficient representation learning and invariant risk minimization, offering a robust solution to the challenges faced by current transfer learning methodologies. The interest in transfer learning across various settings has grown significantly in recent years. Many existing studies have focused on developing transfer learning methods and theories within parametric modeling frameworks. For instance, Bastani 2021 investigates transfer learning in high-dimensional linear regression with a large source dataset, providing an upper bound for the estimation error. Similarly, Li et al. 2022 propose a transfer learning method with minimax optimality in high-dimensional sparse linear regression, featuring a data-driven algorithm to exclude dissimilar source datasets. Both studies assume a small difference between the regression functions of the source and target domains as a prerequisite for successful knowledge transfer. Additionally, Gu et al. 2024 introduce angle-based similarity for transfer learning in high-dimensional linear models. In the context of non-parametric models, Cai and Wei 2021 explore transfer learning for non-parametric classification, defining similarity through the relative signal exponent of the regression function. Cai and Pu 2022 examine transfer learning in non-parametric regression models. These studies require that the models used for both the source and target domains share the same functional form, with similar model parameters across these domains. Knowledge transfer is achieved by transferring parameter estimates from the source domains to the target domain. However, the assumption of having the same model form across domains can be restrictive and may not hold in many real-world scenarios. For example, the source domain might involve a classification problem for cancer sub-types, while the target domain focuses on prognosis prediction for the same cancer. Transferring information from the classification outcomes to a continuous prognosis prediction task presents significant challenges. This limitation highlights the need for more flexible and robust transfer learning approaches that can accommodate varying model forms and complex relationships between domains. It has been suggested that features with strong predictive power offer a significant advantage in transfer learning ( Bengio et al. 2013 ; Neyshabur et al. 2020 ) . Building on this idea, some existing works on transfer learning assume that all tasks share a common representation ( Maurer et al. 2016 ; Arjovsky et al. 2019 ; Hu et al. 2022 ) . Under this shared representation assumption, features learned from the source data can be directly applied to the target task, facilitating efficient knowledge transfer. However, in practice, representations from related tasks are often similar but not identical ( Cai and Wei 2021 ) . Beyond the condition of shared representations, it is important to investigate the role of sufficient representations in facilitating knowledge transfer for general supervised learning problems. Recently, several authors have developed nonlinear representation learning methods based on sufficient dimension reduction ( Lee et al. 2013 ; Huang et al. 2024 ; Chen et al. 2024 ) . Jiao et al. 2024 consider transferring source domain sufficient and domain-invariant representations to the target task. They develop the prediction model for the target task, assuming a linear relationship between the transferred source domain representations and the response of interest. Despite these advancements, the study of data representations in transfer learning problems remains under-explored. Specifically, it remains unclear how to construct representations that can leverage knowledge from source domains while simultaneously capturing specific information from the target domain for general supervised learning problems. In this paper, we introduce a novel transfer learning method that facilitates knowledge transfer through data representation. Specifically, the proposed method first estimates a sufficient and invariant representation from the source domains. This representation is then enhanced with additional independent representations obtained from the target data, which allow us to adapt to the specific characteristics of the target domain. This enhancement strategy ensures its sufficiency for the target domain and still leverages valuable information from the source domains. For simplicity, we refer to the proposed method as Transfer learning through Enhanced Sufficient Representation (TESR). A notable advantage of TESR is that it does not rely on assuming similar model structures across different tasks. For example, the source domain models can be regression models, while the target domain task can be classification. This flexibility makes TESR applicable to a wide range of supervised learning problems where the source and target domains may differ substantially. Additionally, unlike conventional transfer learning approaches, which typically transfer model parameters, a key feature of TESR is that it focuses on transferring data representations from sources to the target domain. Another important aspect of TESR is that it does not assume that a representation sufficient for the source domain will also be sufficient for the target domain. Instead, we posit that a sufficient representation in the source domains provides useful information for the target domain. This assumption allows for greater flexibility and adaptability in the transfer learning. This paper is structured as follows: Section 2 presents the framework of the proposed TESR method.

Read the complete text