W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models
Shang Wang · 2025
Read W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models by Shang Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15983.txt.
Introduction
Small word tools for lean gear are oft found by school hunts that cost more than the tool is worth. Shang Wang hunts with no school at all in a 2025 study. Two marks judge each build in the hunt: its weight count, and the count of chief axes in its mid sheet's weights that bear the bulk of their spread.
The hunt needs no slope sums, so each judge is swift. On two word test sets, the hunt beat schooled hunts on end marks at a scrap of their time. On a bent build space, its rank of builds tracked true worth past the slope borne zero shot hunts. Wang, the sole author, pitches the pair of marks as a lean yard.
Catalog record
| CREATOR | Shang Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15983
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models Shang Wang Abstract The demand for efficient natural language processing (NLP) systems has led to the development of lightweight language models. Previous work in this area has primarily focused on manual design or training-based neural architecture search (NAS) methods. Recently, zero-shot NAS methods have been proposed for evaluating language models without the need for training. However, prevailing approaches to zero-shot NAS often face challenges such as biased evaluation metrics and computational inefficiencies. In this paper, we introduce weight-weighted PCA (W-PCA), a novel zero-shot NAS method specifically tailored for lightweight language models. Our approach utilizes two evaluation proxies: the parameter count and the number of principal components with cumulative contribution exceeding η \eta in the feed-forward neural (FFN) layer. Additionally, by eliminating the need for gradient computations, we optimize the evaluation time, thus enhancing the efficiency of designing and evaluating lightweight language models. We conduct a comparative analysis on the GLUE and SQuAD datasets to evaluate our approach. The results demonstrate that our method significantly reduces training time compared to one-shot NAS methods and achieves higher scores in the testing phase compared to previous state-of-the-art training-based methods. Furthermore, we perform ranking evaluations on a dataset sampled from the FlexiBERT search space. Our approach exhibits superior ranking correlation and further reduces solving time compared to other zero-shot NAS methods that require gradient computation. 1 Introduction Figure 1: Comparison of the running time between W-PCA and other training-based NAS methods for lightweight language models. Our method achieves a substantial reduction in search time for the optimal network structure by two to three orders of magnitude, as we do not need to train the supernet. Large language models (LLMs) have shown exceptional performance across various domains ( OpenAI 2023 ) . However, their size and computational demands pose challenges in resource-constrained environments like mobile devices and edge computing. Therefore, there is a growing need to explore lightweight language models that can operate efficiently on these platforms. One approach to address this challenge is through knowledge distillation (KD) ( Liu et al. 2023 ; Li et al. 2023c ; Li & Jin 2022 ; Li et al. 2023b ; Li 2022 ; Li et al. 2024a ) , where a larger language model acts as a teacher to train a smaller, more lightweight language model ( Turc et al. 2019 ; Sanh et al. 2020 ; Jiao et al. 2020 ; Sun et al. 2020 ; Wang et al. 2020 ) . However, the student models trained for these tasks were manually designed. To effectively search for student models, the use of neural architecture search (NAS) has become essential. NAS is a technique that automates the process of designing neural networks, enabling the exploration of a wide range of architectures to identify the most optimal ones for a given task. Vanilla NAS approaches primarily used reinforcement learning ( Zoph & Le 2016 ) or genetic algorithms ( Real et al. 2019 ) to train neural networks from scratch, but these methods were computationally expensive. Subsequently, one-shot NAS methods, such as gradient-based ( Liu et al. 2018 ) and single path one-shot (SPOS) methods ( Guo et al. 2020 ) , were proposed. These methods are more efficient as they leverage pre-trained models or parameter sharing, requiring the establishment of a supernet in advance from which to sample the optimal subnetworks. Importantly, many lightweight model search tasks in natural language understanding (NLU) are accomplished using one-shot NAS ( Xu et al. 2021 ; Dong et al. 2021 ; Gao et al. 2022 ) . While one-shot NAS reduces training costs compared to training from scratch, it still requires various training strategies to effectively train the supernet. However, to further enhance search efficiency, it is necessary to introduce zero-shot NAS ( Mellor et al. 2021a ) . Zero-shot, also known as training-free NAS, is a promising approach that eliminates the need for training neural networks and directly evaluates their performance using proxy metrics. This significantly reduces the training time and computational resources required for NAS. Existing zero-shot NAS methods ( Abdelfattah et al. 2020 ; Wei et al. 2024 ; Dong et al. 2023a ; Dong et al. 2023b ) have primarily focused on ranking correlations on NAS benchmark datasets ( Klyuchnikov et al. 2022 ) , with limited consideration for specific deep learning tasks. This limitation hinders their applicability and effectiveness in practical scenarios. Additionally, these methods often solely consider a single feature of the models, leading to biased evaluations and potentially overlooking important characteristics. In our research, we aim to address these limitations and improve the applicability of zero-shot NAS. In Section 5 , we conducted ranking correlation experiments. As illustrated in Figure 2 , our attempts to incorporate previous zero-shot proxies into language model evaluation yielded unsatisfactory results. However, we observed a strong correlation in ranking between principal component analysis (PCA) and the number of parameters (#params), with their product demonstrating even better performance. Motivated by these findings, we propose a novel approach called Weight-Weighted PCA (W-PCA), which takes into account both the parameter count and the number of principal components with cumulative contribution exceeding a given \eta threshold in the model. By integrating these two factors, our aim is to achieve a more accurate evaluation of language models in the context of zero-shot NAS. Furthermore, we have designed a search space specifically for NLU tasks and applied our designed zero-shot proxies, as well as the previous zero-shot proxies used in Transformer, to this search space. To the best of our knowledge, this is the first work that applies zero-shot NAS to NLU tasks. Figure 2: Plots depicting the evaluation of zero-shot proxy metrics on 500 randomly sampled architectures from the FlexiBERT search space. As in literature ( Serianni & Kalita 2023 ) , we use the GLUE score of each neural network as the ground truth and evaluate the performance of each zero-shot proxy metric based on its ranking correlation with the ground truth. The specific calculations of PCA is described in Section 3.2 , and the respective zero-shot proxies used for the comparison are summarized in Section 2.3 . Our metric W-PCA is calculated as the product of the number of parameters (#params) and the principal component analysis (PCA). 2 Related Work 2.1 Lightweight BERT Models Turc et al. 2019 observed that distillation and pre-training + fine-tuning have mutually reinforcing effects. DistilBERT ( Sanh et al. 2020 ) utilizes a triple loss function for training the lightweight model. TinyBERT ( Jiao et al. 2020 ) applies distillation in both the pre-training and task-specific learning phases. MobileBERT ( Sun et al. 2020 ) proposes a bottleneck structure to reduce the parameter count. MiniLM ( Wang et al. 2020 ) introduces a compression method called deep self-attentive distillation. In this study, we incorporate both the standard BERT-base ( Devlin et al. 2019 ) and MobileBERT models, along with their weight-sharing variations, where each layer is integrated into the supernet. 2.2 One-shot NAS for Efficient Models Numerous methods have been proposed for performing neural architecture search (NAS) ( Hu et al. 2021 ; Dong et al. 2022 ; Sun et al. 2024 ; Li