Domain Pre-training Impact on Representations

Cesar Gonzalez-Gutierrez, Ariadna Quattoni · 2025

Read Domain Pre-training Impact on Representations by Cesar Gonzalez-Gutierrez, Ariadna Quattoni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24455.txt.

Introduction

Cesar Gonzalez-Gutierrez and Ariadna Quattoni present a 2025 empirical study on how the choice of pretraining text shapes what a transformer model learns. Representation quality is usually credited to model size and data scale. The authors isolate the effect of the corpus itself. Their experiments show that pretraining on a small, specialized corpus can still yield effective representations for matching tasks.

Mixing a generic corpus with a specialized one helped only when the target task's distribution resembled the specialized text. Where the two diverged, the blend lost its edge. The lesson is that similarity, not just volume, decides whether domain pretraining pays. The study speaks to teams building models for narrow fields on modest budgets. It suggests a small, well-matched corpus can beat a bigger mismatched one.

Catalog record

CREATORCesar Gonzalez-Gutierrez, Ariadna Quattoni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24455

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Domain Pre-training Impact on Representations Cesar Gonzalez-Gutierrez, Ariadna Quattoni Abstract This empirical study analyzes the effects of the pre-training corpus on the quality of learned transformer representations. We focus on the representation quality induced solely through pre-training. Our experiments show that pre-training on a small, specialized corpus can yield effective representations, and that the success of combining a generic and a specialized corpus depends on the distributional similarity between the target task and the specialized corpus. 1 Introduction Since 2018, pre-training (PT) has become a standard step in model development, demonstrating effective transfer learning for diverse natural language understanding tasks ( Peters et al., 2018 ; Devlin et al., 2019 ; Radford et al., 2018 ) . This approach leverages textual corpora to induce useful representations by minimizing a self-supervised language modeling loss function. This representation is usually leveraged via transfer learning for downstream tasks using supervised data. Low-Annotation Probing (ALC%) Task Alignment (THAS%) Figure 1: Cross-domain representation performance. We report absolute performance for GD embeddings (center), and relative improvement for ID and DA. When selecting a PT corpus, we generally have three options: 1) use a large, generic corpus \bm{G} , similar to training a foundational model; 2) use a smaller, specialized corpus \bm{S} , which is expected to be more relevant to the target task; 3) combine both \bm{G} + \bm{S} , as in domain-adaptive PT ( Gururangan et al., 2020 ) , where a model initially pre-trained on \bm{G} is further refined using \bm{S} . For example, when developing a toxicity filter for a new forum with only a few labeled comments, one option is to use a pre-trained representation computed on a generic corpus \bm{G} (such as BERT). Alternatively, we could construct a specialized corpus \bm{S} by collecting posts from the forum itself or from other related sources. We can then pre-train using either \bm{G} alone or a combination o both \bm{G} + \bm{S} . We refer to the textual distribution from which a PT corpus is sampled as its domain . The goal of this paper is to understand how pre-training corpus’s choice affects the quality of the learned representation. We focus on measuring representation quality after self-supervised pre-training, without subsequent supervised training (as in fine-tuning). Following prior work, we use three representation quality metrics: a standard probing technique ( Ettinger et al., 2016 ; Adi et al., 2017 ) , and two label-representation alignment metrics, one based on hierarchical clustering structures ( Gonzalez-Gutierrez et al., 2023 ) , the other one based on data-dependent complexity ( Yauney and Mimno, 2021 ) . We conduct an empirical analysis using a wide range of data sources and tasks, evaluating representation quality under different pre-training scenarios. We derive the following conclusions: 1) If the specialized corpus \bm{S} is close to the target task distribution, and it is not too small, pre-training on \bm{S} can be as effective as pre-training on \bm{G} . 2) Pre-training on both \bm{G} and \bm{S} (domain-adaptive pre-training) can sometimes improve performance compared to using \bm{G} alone, but it can also lead to decreased performance. 3) The success of the domain-adaptive strategy depends on the similarity between the target task distribution and that of \bm{S} . We validate this claim by showing a consistent correlation between distributional similarity and domain-adaptive performance improvement, using two different similarity measures. 2 Pre-training Effect on Transformer Embeddings We begin this section with a description of the experimental setting followed by a discussion of the main results. 2.1 Experimental Setting We consider three pre-training scenarios : • In-domain ( ID ): Pre-training a model from scratch on a domain-specific corpus \bm{S} . • General domain ( GD ): The common PLM training approach of using a large generic corpus \bm{G} . • Domain-adaptive ( DA ): Starting with a PLM trained on \bm{G} , continuing pre-training on \bm{S} . To simulate domain-specific pre-training corpora ( \bm{S}_{\text{domain}} ), we used the unlabeled text in the following classification datasets: IMDb ( Maas et al., 2011 , ) , Sentiment 140 ( Go et al., 2009 , ) , Wiki Toxic ( Wulczyn et al., 2017 , ) , Civil Comments ( Borkan et al., 2019 , ) , TREC ( Li and Roth, 2002 ; Hovy et al., 2001 ) , AG-News ( Zhang et al., 2015 ) , and Ohsumed ( Hersh et al., 1994 ) . Test partitions are excluded as held-out sets and are used to compute the representation quality metrics. As a general corpus \bm{G} , we employ a model pre-trained on BERT corpus (BookCorpus plus English Wikipedia). Table 0 shows PT corpus sizes. Corpus Task / Text #words IMDb sentiment / movie reviews 18M Sentiment 140 sentiment / tweets 21M Wiki Toxic toxicity / user discussion 11M Civil Comments toxicity / user comments 100M TREC topic / short questions 56k AG-News topic / news 4.5M Ohsumed topic / medical abstracts 1.8M BookCorpus – / books 800M Wikipedia (EN) – / encyclopedia 2.5B Table 1: Pre-training corpora. We choose a BERT model architecture ( Devlin et al., 2019 ) because it is well studied transformer model ( Rogers et al., 2020 ) and has a size suitable for running multiple pre-training experiments. Following Liu et al. (2019b) , we omit the next sentence prediction (NSP) task and focus on masked-LM (MLM). To obtain sentence embeddings , we extract mean token embeddings from the last layer. In A.6 , we employ other representation functions to explore the roles of different layers and tokens. To quantify the changes underwent by representations during PT, we employ three evaluation metrics targeted at the model’s embedding space quality w.r.t. a task: probing and two label-representation alignment scores. In A.4 , we explore an intrinsic clustering quality metric that does not depend on task labels. Probing ( Ettinger et al., 2016 ; Adi et al., 2017 ) uses weak classifiers to evaluate the task performance attributable to the representation. We use low-annotation probes to test the representation’s ability to uncover structures that enable learning from few samples. In particular, the probes are MaxEnt classifiers on top of the embeddings trained with sample sizes ranging from 100 to 1000, increasing in steps of 100. We report the area under this learning curve (ALC). Task Hierarchical Alignment Score ( Gonzalez-Gutierrez et al., 2023 , THAS;) quantifies the alignment between representation hierarchical clustering structures and task labels. This metric measures the degree of cluster purity at different hierarchical levels. A representation capable of perfectly separating n classes into n pure clusters will obtain the maximum score. More precisely, we used agglomerative clustering on the embeddings and, for each partition, measured the area under the precision-recall curve using in-cluster class prevalence as label predictions for each data point. Data-Dependent Complexity ( Yauney and Mimno, 2021 , DDC;) quantifies the compatibility between a representation and a binary classification task. It captures patterns through the eigen decomposition of a kernel matrix that measures sample similarity in the representation space. Label alignment is evaluated based on the extent to which the label vectors can be reconstructed from their projections onto the top eigenvectors of the kernel matrix. These three metrics are computed on the test partitions of the tasks associated

Read the complete text