MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning
Vera Pavlova, Mohammed Makhlouf · 2025
Read MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning by Vera Pavlova, Mohammed Makhlouf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16797.txt.
Introduction
Vera Pavlova and Mohammed Makhlouf, a two-author team, adapt general text embedding models to specialized fields with a framework called MOSAIC. Embeddings turn text into numbers so machines can judge similarity. Models trained on general text often miss the vocabulary and sense of a narrow domain, yet retraining from scratch is costly.
MOSAIC trains with two goals at once: masked language modeling, which teaches domain wording, and contrastive learning, which keeps meanings apart. The stages are balanced and applied in sequence. Tests on rich and sparse domains show clear gains in ranking quality over strong general baselines, with improvements up to about thirteen percent on a standard top-ten measure. Ablations confirm each stage earns its place.
Catalog record
| CREATOR | Vera Pavlova, Mohammed Makhlouf |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-16797
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning Vera Pavlova, Mohammed Makhlouf Abstract We introduce MOSAIC ( M asked O bjective with S elective A daptation for I n-domain C ontrastive learning), a multi-stage framework for domain adaptation of sentence embedding models that incorporates joint domain-specific masked supervision. Our approach addresses the challenges of adapting large-scale general-domain sentence embedding models to specialized domains. By jointly optimizing masked language modeling (MLM) and contrastive objectives within a unified training pipeline, our method enables effective learning of domain-relevant representations while preserving the robust semantic discrimination properties of the original model. We empirically validate our approach on both high-resource and low-resource domains, achieving improvements up to 13.4% in NDCG@10 (Normalized Discounted Cumulative Gain) over strong general-domain baselines. Comprehensive ablation studies further demonstrate the effectiveness of each component, highlighting the importance of balanced joint supervision and staged adaptation. 1 Introduction Self-supervised learning has enabled significant progress in natural language processing, with methods like MLM ( Devlin et al. 2019 ; Liu et al. 2020 ; Conneau et al. 2020 ; Sanh et al. 2019 ) and contrastive training ( Reimers and Gurevych 2019 ; Wu et al. 2020 ; Liu et al. 2021 ) driving recent developments. However, these methods are typically explored separately, as effectively combining MLM and contrastive learning remains a significant challenge, since their joint optimization often results in conflicting training signals and suboptimal performance ( Gao et al. 2021 ) . Nevertheless, unifying these objectives presents an opportunity to leverage the complementary strengths of token-level (MLM) and sentence-level (contrastive) supervision, while also improving the quality of learned representations by mitigating the anisotropy problem (a phenomenon that confines embeddings to a narrow cone-like region in the vector space, thereby limiting their expressiveness) ( Ethayarajh 2019 ; Li et al. 2020 ; Gao et al. 2021 ) . While there have been successful attempts to combine MLM and contrastive objectives for training language models ( Meng et al. 2021 ; Chi et al. 2021 ) and sentence embeddings ( Gao et al. 2021 ; Wu et al. 2022 ; Giorgi et al. 2021 ) , the majority of the prior work has focused on general-domain data. General-domain sentence embedding models are now widely available, many trained on vast general-domain corpora using a two-stage approach: an initial pre-training phase on massive unlabeled data, followed by supervised fine-tuning ( Wang et al. 2022b ; Li et al. 2023 ; Nussbaum et al. 2024 ; Merrick et al. 2024 ) . The data used for pre-training can exceed half a billion sentence pairs (hundreds of gigabytes of text), resources that are rarely available in specific domains. Although these general-domain models can perform competitively in specialized areas, their lack of domain-specific knowledge often limits performance. To address this gap, we propose domain adaptation of pre-trained embedding models that leverage their ability to distinguish between similar and dissimilar pairs and transfer it to a domain-specific embedding model. Previous research on language model adaptation highlights the importance of domain-specific vocabulary for improving results on downstream tasks ( Beltagy et al. 2019 ; Gu et al. 2020a ) . However, simply adding domain-specific vocabulary and continuing MLM training degrades the contrastive properties of the learned representations, since the encoder loses its desirable characteristics under the token prediction objective ( Wu et al. 2022 ) . On the other hand, adding new tokens and continuing only with the contrastive objective provides insufficient training signals to update new domain tokens, as the embedding matrix receives diluted signals due to the pooling functions applied to generate sentence embeddings. This dilemma motivates our approach of using a joint objective to enable both token-level and sentence-level supervision, thus benefiting from both worlds and enhancing domain adaptation for both the encoder and the embedding matrix during training. Building on a mutual information maximization perspective ( Hjelm et al. 2018 ; Bachman et al. 2019 ; Kong et al. 2019 ; Chen et al. 2020 ; Chi et al. 2021 ) , which demonstrates that these objectives are aligned rather than contradictory, operating at different levels of language granularity, we leverage the joint optimization of MLM and contrastive objectives. Though these objectives are theoretically aligned, a key challenge in joint training arises from the dominance of the MLM loss and more frequent token-level supervision, which can overwhelm the joint objective and hinder balanced optimization. We mitigate this issue by directing the MLM signal toward domain-relevant tokens, rather than separating encoders ( Wu et al. 2022 ) , which limits the flow of informative token-level supervision into sentence-level embeddings. Importantly, MOSAIC builds on a very simple prerequisite: vocabulary expansion. The method only requires adding new domain tokens to the model’s tokenizer, which is a cheap and practical step that avoids complex architectural changes or large-scale retraining. This simplicity makes MOSAIC broadly applicable across domains. Figure 1: The MOSAIC pipeline: a multi-stage framework for domain adaptation of sentence embedding models. To thoroughly evaluate our method, we apply it to both high-resource and low-resource domains. Most domain-adaptation research focuses on high-resource and medium-resource domains, which is valuable for benchmarking, comparison with strong baselines, and conducting ablation studies. Yet this focus restricts the generalizability of adaptation methods to truly low-resource domains, which are common in real-world applications. Such domains often face acute data scarcity, making robust adaptation methods essential for ensuring equitable access to state-of-the-art language technologies and maximizing the real-world impact of embedding models. To demonstrate the robustness and practical value of our approach, we validate it in two domains: the Biomedical domain, which is characterized by high-resource scientific texts, and the Islamic domain, which represents low-resource but culturally significant content. This allows us to test the robustness of our method even when there is very limited in-domain data. Our main contributions are as follows: (1) We propose MOSAIC, a novel domain adaptation approach for pretrained sentence embedding models that jointly optimizes MLM and contrastive objectives within a mutual information maximization framework. (2) We empirically validate our method on both high-resource (biomedical) and low-resource (Islamic) domains, demonstrating substantial gains over strong general-domain baselines. (3) We conduct comprehensive ablation studies to analyze the contribution of each component and the dynamics of joint objective training. (4) We release our code and pretrained models to support reproducibility and facilitate future research. 2 Related Work 2.1 Contrastive and MLM Objectives Contrastive Predictive Coding (CPC) introduced the InfoNCE loss ( van den Oord et al. 2018 ) , encouraging representations to align with an anchor while separating from negatives, thereby laying the groundwork for contrastive learning. In parallel, Masked Language Modeling (MLM), introduced with BERT ( Devlin et al. 2019 ) , emerged as the standard pretraining objective for encoder-based models. Prior work has explored joint contrastive and masked objectives in two distinct directions: large-scale language model pretraining and general-purpose sentence embedding models. COCO-LM integrates contrastive learning into a pretraining pipeline for transformer language models ( Vaswani et al. 2017 ; Meng et al. 2021 ) , replacing BERT’s Next Sentence Prediction (NSP) objective with a more effective contrastive signal by pairing corrupted and truncated versions of a sentence. The model jointly learns to align these pairs via contrastive loss and to correct