Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Models

Haomin Qi 1 , Chengbo Huang 2 , Zihan Dai 3 , Yunkai Gao 4 · 2025

Read Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Models by Haomin Qi 1 , Chengbo Huang 2 , Zihan Dai 3 , Yunkai Gao 4 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17344.txt.

Introduction

Haomin Qi and three co-authors tune large language models for many languages at once, including ones with little training text. Their method mixes two cheap tuning styles layer by layer and locks some layers to rotations that keep training stable. Alongside, simple data housekeeping — spotting each text's language, dropping near-duplicates, and filtering junk — guards the training set.

On the XNLI and FLORES test sets, the hybrid beat strong low-cost tuning baselines. It also kept answers balanced across languages and made the model's confidence scores more honest. Training cost stayed modest. One caution for readers: the arXiv record carries an admin note flagging substantial text overlap with an earlier paper, arXiv:2507.18076.

Catalog record

CREATORHaomin Qi 1 , Chengbo Huang 2 , Zihan Dai 3 , Yunkai Gao 4
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-17344

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Models Haomin Qi 1 , Chengbo Huang 2 , Zihan Dai 3 , Yunkai Gao 4 Abstract We present a governance-aware hybrid fine-tuning framework for multilingual, low-resource adaptation of large language models. The core algorithm mixes gradient-aligned low-rank updates with structured orthogonal transformations through layer-wise mixing and introduces unitary constraints in selected sub-layers to stabilize deep optimization. In tandem with lightweight, label-free data governance—language identification, near-duplicate removal, and quality filtering—the framework targets accuracy, calibration, and cross-language parity under tight compute budgets. Across XNLI and FLORES, the hybrid approach delivers consistent gains over strong PEFT baselines while maintaining directional balance and improving probability calibration, as shown in Tables II and III . It is more resilient to lightweight orthographic variants, as shown in Table IV , and benefits additively from simple governance steps, as shown in Table V . Training-footprint measurements indicate modest overhead and a favorable cost–quality frontier, as shown in Table VI and Figure 2 . Together these results show that hybrid and unitary PEFT provide a stable and accessible path to resource-efficient multilingual adaptation when paired with practical data governance. I Introduction Large Language Models (LLMs) have become central to Natural Language Processing (NLP) applications ranging from machine translation to code generation, yet full-model adaptation is often prohibitive in memory and time. Parameter-Efficient Fine-Tuning (PEFT) mitigates this by updating a small subset of parameters via lightweight adapters or structured transforms, with notable instances including Low-Rank Adaptation (LoRA) [ 1 ] , Orthogonal/OFT-style updates refined as Butterfly Orthogonal Fine-Tuning (BOFT) [ 2 ] , and gradient-aligned variants that steer updates along dominant directions [ 3 ] . Despite their efficiency, existing PEFT approaches trade off rapid early adaptation, stability in deep stacks, and representational capacity, particularly when supervision is scarce or distributional shifts induce fragile optimization. We propose a hybrid PEFT framework that fuses gradient-aligned low-rank updates with structured orthogonal adjustments through layer-wise, gradient-norm–based mixing. Concretely, the low-rank branch (LoRA-style with gradient alignment) accelerates early progress by emphasizing dominant update directions, while the orthogonal branch (BOFT-style Cayley/orthogonal updates) preserves gradient magnitudes and encourages stable late-phase optimization [ 4 ] . To further steady very deep Transformer stacks, we impose selective unitary constraints in chosen sub-layers using structured unitary parameterizations inspired by Unitary Recurrent Neural Networks (uRNNs) [ 5 ] . The result is a single-step, per-layer fusion that retains PEFT simplicity while improving stability and invariance at modest overhead; the mechanism is compatible with standard training loops and does not require altering the backbone architecture. Our evaluation spans general-purpose and multilingual, low-resource settings across models from 7B to 405B parameters. We report accuracy and Expected Calibration Error (ECE) [ 6 ] on standard benchmarks—GLUE [ 7 ] , GSM8K [ 8 ] , MT-Bench [ 9 ] , and HumanEval [ 10 ] —and study cross-language accuracy and parity on XNLI [ 11 ] and FLORES [ 12 ] under a 32-shot protocol. Because deployability depends on how models are evaluated and curated, we complement accuracy with cross-language parity (Parity Gap, \Delta ) and calibration, probe robustness to routine orthographic variants, and quantify the effect of lightweight, label-free curation (language identification and near-duplicate/quality filtering) using widely adopted tools and heuristics [ 13 , 14 ] . Empirically, the hybrid method tracks or narrows the gap to full fine-tuning with far fewer trainable parameters, improves calibration and parity under limited supervision, exhibits smaller drops under orthographic variation, and remains near a favorable cost–quality frontier, as detailed in Tables I – VI and Figures 2 – 4 . We summarize the contributions of our paper as follows: 1) Hybrid PEFT. A layer-wise fusion of gradient-aligned low-rank and structured orthogonal updates via gradient-norm mixing, yielding fast early adaptation and stable late optimization under a fixed parameter budget. 2) Unitary stabilization. Selective unitary parameterizations for key Transformer sub-layers that control gradient growth in deep stacks with negligible overhead. 3) Validated effectiveness. Consistent gains over strong PEFT baselines from 7B to 405B on general-purpose and 32-shot multilingual tasks, with improved calibration, cross-language parity, and robustness to orthographic variants. II Related Work II-A Parameter-Efficient Fine-Tuning (PEFT) Parameter-Efficient Fine-Tuning (PEFT) reduces the number of trainable parameters for Large Language Models (LLMs) in Natural Language Processing (NLP) by freezing most pre-trained weights and learning small task-specific modules. Early approaches include adapters , which insert bottleneck layers in Transformer blocks [ 15 ] , and prefix/prompt tuning , which optimizes continuous prompts while keeping backbone parameters fixed [ 16 , 17 ] . Subsequent methods improved parameter sharing and compression, e.g., Compacter [ 18 ] , and even showed that updating only biases ( BitFit ) can be competitive for many tasks [ 19 ] . These techniques collectively emphasize modularity, small memory footprints, and ease of deployment in constrained environments. Low-Rank Adaptation (LoRA) decomposes weight updates into low-rank matrices, offering a favorable accuracy–memory trade-off and becoming a widely used PEFT baseline. QLoRA combines PEFT with 4-bit quantization to further reduce memory while preserving headroom for quality [ 20 ] . In parallel, gradient-aligned schemes project or initialize updates along dominant gradient subspaces to improve early progress and memory use. Our work follows this line by combining a gradient-aligned low-rank branch with a structured orthogonal branch under a fixed PEFT budget, aiming to retain rapid adaptation while strengthening stability in deeper layers. Whereas prior PEFT methods typically commit to a single update family (low-rank, orthogonal, prompts/adapters) or a single stabilization strategy, we combine gradient-aligned low-rank and structured orthogonal updates through a layer-wise mixing rule, and we introduce selective unitary constraints to steady deep stacks. II-B Orthogonal and Unitary Parameterizations Orthogonal and unitary parameterizations preserve norms and have long been used to stabilize deep optimization. In recurrent networks, unitary/orthogonal transitions alleviate vanishing and exploding gradients [ 5 , 21 ] . In the PEFT setting, Butterfly Orthogonal Fine-Tuning (BOFT) constrains updates via butterfly factorization, linking to fast linear transforms and orthogonal structure [ 2 , 22 ] . Rather than replacing low-rank updates, we fuse orthogonal and low-rank branches per layer and add selective unitary constraints to key Transformer sub-layers, seeking a complementary balance: the low-rank path emphasizes data-aligned efficiency, while the orthogonal path encourages stable long-horizon optimization in deep stacks. II-C Multilingual Adaptation, Calibration, and Robustness Multilingual evaluation emphasizes cross-language generalization and parity across families and scripts. XNLI benchmarks cross-lingual natural language inference [ 11 ] ; FLORES-101/200 measures translation quality across many languages [ 12 , 23 ] . Beyond accuracy, Expected Calibration Error (ECE) captures the reliability of predicted probabilities in modern neural networks. Robustness to lightweight orthographic variation (e.g., punctuation normalization, diacritics removal, whitespace changes) is relevant for user-facing systems; small character-level perturbations can degrade performance in classification and translation [ 24 , 25 ] . Our experimental protocol aligns with this perspective by reporting accuracy with Parity Gap ( \Delta ), ECE, and sensitivity to orthographic variants under a matched few-shot budget. II-D Data Governance and Budget-Aware Adaptation Data governance practices such as language identification, near-duplicate removal, and light quality filtering are standard in web-scale pipelines and benefit multilingual robustness [ 26 ] . These steps are complementary to PEFT and quantization when compute is constrained. Our

Read the complete text