Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning
Mohamed Hesham Ibrahim Abdalla Department of Computer Science University of Technology Nuremberg [email protected] &Zhipin Wang Department of Computer Science University of Technology Nuremberg [email protected] &Christian Frey Department of Computer Science University of Technology Nuremberg c · 2025
Read Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning by Mohamed Hesham Ibrahim Abdalla Department of Computer Science University of Technology Nuremberg [email protected] &Zhipin Wang Department of Computer Science University of Technology Nuremberg [email protected] &Christian Frey Department of Computer Science University of Technology Nuremberg c (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19733.txt.
Introduction
M. H. I. Abdalla and four co-authors present Zhyper, an efficient way to steer a large language model toward a stated culture, political outlook, or other text-specified conditioning. Prompting alone cannot guarantee the model behaves as asked, because pretraining biases push back. Earlier fixes fine-tuned LoRA adapter weights directly for each conditioning, at a large parameter cost.
Zhyper instead uses a factorized hypernetwork that generates context-aware LoRA adapters from a written description of the desired conditioning. On multiple benchmarks it matched leading baselines while using up to twenty-six times fewer parameters. Extended to cultural alignment, it generalized better to settings outside its training domain and captured finer-grained contextual values.
Catalog record
| CREATOR | Mohamed Hesham Ibrahim Abdalla Department of Computer Science University of Technology Nuremberg [email protected] &Zhipin Wang Department of Computer Science University of Technology Nuremberg [email protected] &Christian Frey Department of Computer Science University of Technology Nuremberg c |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19733
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning Mohamed Hesham Ibrahim Abdalla Department of Computer Science University of Technology Nuremberg [email protected] &Zhipin Wang Department of Computer Science University of Technology Nuremberg [email protected] &Christian Frey Department of Computer Science University of Technology Nuremberg [email protected] &Steffen Eger Department of Computer Science University of Technology Nuremberg [email protected] and Josif Grabocka Department of Computer Science University of Technology Nuremberg [email protected] Abstract Large Language Model (LLM) conditioning refers to instructing an LLM to generate content in accordance with the norms and values of a specific culture, beliefs of a particular political orientation, or any desired text-specified semantic conditioning. Unfortunately, prompt engineering does not ensure that LLMs behave in accordance with a desired conditioning due to the inductive bias of the pre-training and alignment datasets. Prior works have focused on fine-tuning LLMs by directly conditioning the LoRA weights; however, such methods introduce a large number of parameters. As a remedy, we propose Zhyper, a parameter-efficient factori z ed hyper network framework that generates context-aware LoRA adapters from textual descriptions. Experiments on multiple benchmarks show that Zhyper achieves competitive performance with up to 26x fewer parameters than the state-of-the-art baselines. Furthermore, we extend Zhyper to cultural alignment, demonstrating improved generalization to out-of-domain settings and a better capturing of fine-grained contextual values. 1 Introduction Large Language Models (LLMs) have transformed Natural Language Processing (NLP), Computer Vision (CV), and machine learning (ML) more broadly. They achieve state-of-the-art performance in text generation and comprehension across diverse domains, including code synthesis ( Rozière et al. 2023 ) , mathematical reasoning ( Ahn et al. 2024 ) , scientific writing ( Geng et al. 2025 ; Eger et al. 2025 ) , multimodal tasks such as text–image understanding and generation ( Alayrac et al. 2022 ) , and evaluation of machine translation and related tasks ( Gu et al. 2025 ) . This success stems from scaling to millions and billions of parameters. However, this scaling requires large computational resources, motivating the search for parameter-efficient fine-tuning (PEFT) techniques. Recent advances have made it possible to adapt LLMs to task-specific criteria, which is crucial for a broader applicability and acceptance of NLP systems. A recent stream of research leverages PEFT techniques ( Ding et al. 2023 ; Weyssow et al. 2023 ; Prottasha et al. 2024 ) , e.g., Low-Rank Adaptions (LoRA) ( Hu et al. 2021 ) to adapt for desired task-specific values in an LLM. LoRA achieves this by freezing most of the pre-trained model’s parameters and introducing trainable low-rank matrices, yielding weight correction terms. However, stand-alone LoRA approaches are primarily tailored for a single-task adaptation and may lose their effectiveness in a setting where an LLM needs to be adapted to various downstream settings. Therefore, approaches directly tackling a multi-task learning (MTL) setting have been proposed ( Agiza et al. 2024 ; Wang et al. 2023 ; Luo et al. 2024 ; Wang et al. 2024 ) that aim to do multi-task fine-tuning efficiently, where a shared backbone model must serve multiple tasks. A promising direction for the dynamic and robust individualization of LLMs is by leveraging hypernetworks in the training pipeline. In Text-to-LoRA (T2L) ( Charakorn et al. 2025 ) , the authors apply hypernetworks to adapt LLMs to specific task descriptions using only a textual task description as the input for learning the adapters’ weights. However, two open challenges remain unresolved. First, existing conditioned LoRA methods, such as T2L, are not parameter-efficient when extended to large contextual spaces. Second, the applicability of conditioned LoRA tuning has not been explored for the important real-world problem of cultural alignment. To tackle the described challenges, we propose a factori z ed hyper network, called Zhyper , which leverages a hypernetwork to inject desired values in the outputs of an LLM. More specifically, the hypernetwork should produce a different weight based on the current layer, a layer type for attention-awareness, and the respective description of a context we want to adapt to. As opposed to prior works ( Charakorn et al. 2025 ) , we additionally experiment with contexts being descriptions of cultures. Considering the example shown in Figure 1 , the goal is to condition a base model on certain criteria. For instance, when choosing a preferred food, the answer might have country-specific dependence. The contextual modulation signal is computed via a hypernetwork that is integrated into the computation of the LoRA adapter, leading to answers conditioned on the instilled values. We empirically show that our novel model achieves comparable predictive performance at an order of magnitude fewer parameters on a variety of LLM capability assessments, e.g., math, science, coding, reasoning, and word knowledge. Furthermore, we provide a thorough ablation study on the contextual modulation signal represented as an (r\times r) -matrix, where r denotes the rank of the LoRA adapters. Figure 1: We introduce a novel parameter-efficient architecture for conditioned LLM finetuning based on hypernetwork-conditioned LoRA adapters Our contributions are as follows: • A novel lightweight hypernetwork-based architecture for training LoRA adapters that align to text or culture descriptions with up to 26x fewer parameters compared to prior work. • Hypernetwork that generates a compact contextual modulation signal instead of generating all parameters of an adapter. • A thorough empirical study on efficient learning strategies for the conditioned fine-tuning of Large Language Models. • Improved empirical performances in the important use cases of task conditioning and cultural alignment. 2 Zhyper - Conditioned LLM Tuning Our method Zhyper leverages hypernetworks to induce descriptive information and generate LoRA adapters for context-specific adaptation. The following subsections present the preliminaries in Section 2.1 , our novel factorized architecture in Section 2.2 , and the complexity analysis of our method in Section 2.3 . 2.1 Preliminaries. Low-Rank Adaptation (LoRA) is a well-established parameter-efficient fine-tuning technique for LLMs ( Hu et al. 2021 ) . Generally, the weights of a base model are frozen, and only low-rank weight matrices are trained, serving as adapters to the model. Formally, for each selected linear transformation h={\bm{W}}^{\text{base}}{\bm{x}} , the fine-tuned transformation is given by h^{\prime}={\bm{W}}^{\text{base}}{\bm{x}}+\Delta{\bm{W}}{\bm{x}} , with \Delta{\bm{W}}={\bm{A}}{\bm{B}} , where {\bm{A}}\in\mathbb{R}^{d_{\text{in}}\times r} , and {\bm{B}}\in\mathbb{R}^{r\times d_{\text{out}}} are low-rank weight matrices with r\ll d . Hypernetworks introduce neural networks whose output defines the parameters of another network ( Ha et al. 2016 ) . It formalizes the idea of learning to generate weights rather than learning weights directly. Formally, let f_{\theta}(x) denote a parameterized target network with \theta\in\mathbb{R}^{r} . A parameterized hypernetwork H_{\phi}(z):\mathbb{R}^{m}\rightarrow\mathbb{R}^{r} is conditioned on a layer-specific descriptor vector {\bm{z}}\in\mathbb{R}^{m} and parameters \phi , and outputs the weights \theta of the target network. 2.2 Architecture We present the Zhyper method, a hypernetwork-conditioned low-rank adaptation method that enables parameter-efficient and context-aware fine-tuning of LLMs. The general workflow of our method is illustrated in Figure 1 , where in the following we provide details on the respective components. Contextual Information. We represent contextual features (e.g., value or cultural descriptions) leveraging a transformer-based encoder trained for general text embeddings. Each description is transformed into a fixed-length embedding vector {\bm{c}}\in\mathbb{R}^{d_{c}} , which serves as the contextual input to our hypernetwork described below. This representation ensures that diverse textual descriptions are mapped into a unified semantic space suitable for conditioning LoRA adapters. We denote by