DeFT-X : Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer

Sona Elza Simon, Preethi Jyothi · 2025

Read DeFT-X : Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer by Sona Elza Simon, Preethi Jyothi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15090.txt.

Introduction

Sona Elza Simon and Preethi Jyothi present, in a 2025 paper, a cleaner way to adapt language models to new languages without retraining them whole. Transfer from rich languages to poor ones remains a stubborn gap. A prior approach tuned only sparse subsets of weights, one set for the task and one for the language.

Those sparse sets were picked by simple gradient rules, which let noise in. The new method denoises the selection before fine-tuning. The tuned pieces then compose with the base model for zero-shot use in the target language. Cleaner masks should mean the small tuned fraction carries signal, not static, for languages with little data.

Catalog record

CREATORSona Elza Simon, Preethi Jyothi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15090

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DeFT-X : Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer Sona Elza Simon, Preethi Jyothi Abstract Effective cross-lingual transfer remains a critical challenge in scaling the benefits of large language models from high-resource to low-resource languages. Towards this goal, prior studies have explored many approaches to combine task knowledge from task-specific data in a (high-resource) source language and language knowledge from unlabeled text in a (low-resource) target language. One notable approach proposed composable sparse fine-tuning (SFT) for cross-lingual transfer that learns task-specific and language-specific sparse masks to select a subset of the pretrained model’s parameters that are further fine-tuned. These sparse fine-tuned vectors (SFTs) are subsequently composed with the pretrained model to facilitate zero-shot cross-lingual transfer to a task in a target language, using only task-specific data from a source language. These sparse masks for SFTs were identified using a simple magnitude-based pruning. In our work, we introduce DeFT-X , a novel composable SFT approach that denoises the weight matrices of a pretrained model before magnitude pruning using singular value decomposition, thus yielding more robust SFTs. We evaluate DeFT-X on a diverse set of extremely low-resource languages for sentiment classification (NusaX) and natural language inference (AmericasNLI) and demonstrate that it performs at par or outperforms SFT and other prominent cross-lingual transfer baselines. 1 Introduction Pretrained language models (LMs) are the de-facto choice for NLP, achieving state-of-the-art results across diverse benchmarks. However, effectively adapting these models to specific tasks remains a challenge owing to their large model sizes and the substantial training costs incurred during full fine-tuning. Furthermore, full fine-tuning approaches are prone to issues such as catastrophic forgetting and negative interference when adapted to multiple tasks. To mitigate these challenges, parameter-efficient fine-tuning (PEFT) techniques are a popular choice Pfeiffer et al. 2024 . These approaches include sparse fine-tuning (SFT) that refers to identifying a sparse subnetwork of the full model to train, and adapter-based methods that insert additional trainable modules while keeping the original model parameters fixed Pfeiffer et al. 2020 ; Hu et al. 2022 . Multilingual NLP introduces an additional layer of complexity, especially in the context of low-resource languages. A key objective of recent work in multilingual NLP has been to facilitate cross-lingual transfer by leveraging high-resource language data to improve performance on low-resource languages. Zero-shot cross-lingual transfer refers to the more constrained setting of having access only to task-specific labeled data in a high-resource (source) language and no labeled data in a low-resource (target) language. PEFT-based approaches such as MAD-X ( Pfeiffer et al. 2020 ) and LT-SFT ( Ansell et al. 2022 ) are designed to support zero-shot cross-lingual transfer. In MAD-X, a task adapter is learned using labeled data in a source language and a language adapter is learned using unlabeled data in a target language. To achieve zero-shot cross-lingual transfer, the task adapter in the source language is combined with the language adapter in the target language. LT-SFT (Lottery Ticket Sparse Fine-Tuning) removes the need for new trainable modules as in adapters by first identifying sparse subnetworks (a.k.a. “lottery tickets") within the model using magnitude pruning. Fine-tuning these subnetworks while keeping the rest of the model frozen yields sparse vectors; these vectors can be estimated independently to learn task-specific and language-specific knowledge. These vectors are composed via simple addition to obtain the final model for zero-shot cross-lingual transfer. Figure 1: A graphical representation of DeFT-X . The pretrained model \theta (gray, left) undergoes full fine-tuning to obtain \theta_{\text{FFT}} . The difference \Delta W (blue and red, left) captures the magnitude difference between \theta and \theta_{\text{FFT}} . Each weight matrix in \Delta W is denoised by pruning higher-order components (i.e., lower singular value components) while retaining lower-order components (i.e., high singular value components). The denoised \Delta W is then magnitude-pruned and sparsely fine-tuned to produce \phi . Finally, the language-specific component \phi_{L} and task-specific component \phi_{T} are combined via addition to form the target language-task model \theta_{\text{TL}} (left). In this work, we propose DeFT-X , a novel sparse composable approach for zero-shot cross-lingual transfer. Adopting the LT-SFT template, DeFT-X aims to identify subnetworks for further finetuning using a low-rank approximation and improves the quality of sparse fine-tuned vectors by denoising higher-order components. Specifically, we use Singular Value Decomposition (SVD) ( Sharma et al. 2023 ; Zhao et al. 2025 ) to decompose each weight matrix into lower and higher-order components (corresponding to high and low singular values, respectively). The higher-order components are denoised and added to the SVD lower-order components, resulting in a signal-amplified weight matrix (refer to \lx@sectionsign 3.1 ). Next, we apply magnitude pruning to identify a more effective sparse subnetwork by removing noise, followed by fine-tuning to obtain the final sparse fine-tuned vectors (detailed in \lx@sectionsign 3.1 ). These vectors are finally composed via simple addition for effective cross-lingual transfer (as shown in \lx@sectionsign 3.2 ). An illustration of DeFT-X is shown in Figure 1 . We evaluate the zero-shot cross-lingual performance of DeFT-X on sentiment analysis (SA) and natural language inference (NLI) tasks for extremely low-resource Indonesian/indigenous languages in the NusaX/AmericasNLI benchmarks, respectively. Our results demonstrate that DeFT-X outperforms state-of-the-art baselines. Our findings highlight the importance of each step in the DeFT-X pipeline – denoising the model before selecting a sparse subnetwork, enforcing sparsity after denoising and fine-tuning the subnetwork, particularly in low-resource settings. 2 Background 2.1 Lottery Ticket Hypothesis (LTH) The Lottery Ticket Hypothesis (LTH) ( Frankle and Carbin 2019 ; Malach et al. 2020 ) states that within a randomly initialized, dense neural network, there exists a subnetwork referred to as a winning ticket that when trained in isolation with its original initialization, can achieve comparable or even superior performance to the full network. Simple pruning techniques such as magnitude-based pruning can be used to identify such trainable subnetworks in fully connected and convolutional networks Han et al. 2015 ; Han et al. 2016 ; Yang et al. 2017 . To identify a winning ticket, a neural network f(x;\theta^{(0)}) is first initialized with random parameters \theta^{(0)} . The network is trained for j iterations, resulting in parameters \theta^{(j)} . Next, p % of the smallest-magnitude weights are pruned, creating a binary mask m . Finally, the remaining parameters are reset to their original values from \theta^{(0)} to yield the winning ticket f(x;m\odot\theta^{(0)}) . However, if a winning ticket’s parameters are randomly re-initialized, its performance deteriorates, highlighting the importance of proper initialization for effective training. This process of pruning and resetting helps uncover subnetworks that retain strong learning abilities when trained in isolation. 2.2 Lottery Ticket Sparse Fine-Tuning (LT-SFT) Inspired by the LTH, Ansell et al. 2022 proposed a lottery ticket-based algorithm (LT-SFT) for efficient zero-shot cross-lingual transfer learning for low-resource languages. LT-SFT proposes a parameter-efficient fine-tuning approach that is both modular like adapters ( Pfeiffer et al. 2021a ; Pfeiffer et al. 2020 ) (i.e., it can be easily combined to adapt a model to different knowledge sources) and expressive like sparse fine-tuning (i.e., it changes the behavior of all components). LT-SFT outperformed the state-of-the-art MAD-X adapter-based approach for low-resource cross-lingual transfer; the algorithm is detailed below. LT-SFT: Generate Sparse Fine-tuned Vectors (SFTs). The LT-SFT algorithm generates sparse fine-tuned vectors (SFTs) in two phases: (1) Pretrained model parameters \theta^{(0)}

Read the complete text