Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering

Hao Gong 1 , Ruilin Gong 2 , Yining Huang 3 · 2026

Read Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering by Hao Gong 1 , Ruilin Gong 2 , Yining Huang 3 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31432.txt.

Introduction

Gong and two colleagues present BiRG-LoRA, a rank-gated adapter method for medical question answering. Medical questions span drugs, diagnosis, public health, and nursing, and each kind may need a different low-rank update, while plain recall items should barely touch the base model. One fixed adapter cannot serve all of them well.

BiRG-LoRA keeps one LoRA module per layer but lets each question choose a sparse set of rank atoms. A biaxial gate mixes semantic evidence with specialty and clinical-operation priors, and a scalar coefficient sets the update strength. On a matched Qwen3-8B protocol it posts the best macro-average across CMB, CMExam, MedQA, and MedMCQA among trainable rivals. It also beats a mixture rival while using about 28 percent fewer trainable parameters.

Catalog record

CREATORHao Gong 1 , Ruilin Gong 2 , Yining Huang 3
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31432

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering Hao Gong 1 , Ruilin Gong 2 , Yining Huang 3 Abstract Medical multiple-choice question answering requires parameter-efficient adaptation across heterogeneous knowledge domains and reasoning operations. A medication question, a diagnostic decision, a public-health item, and a nursing-action item may require different low-rank updates, while some recall items should preserve the base model’s representation with only mild adapter intervention. We propose BiRG-LoRA, a single-adapter rank-gated LoRA method for medical question answering. BiRG-LoRA keeps one LoRA module per target layer but makes its rank dimension input-conditioned: for each question, a biaxial gate combines hidden semantic evidence with specialty/profession priors, clinical-operation priors, and their interaction to select a sparse top- k k subset of rank atoms. A scalar injection coefficient further controls the strength of the selected adapter update. Under a matched Qwen3-8B CMB-source protocol, BiRG-LoRA achieves the highest four-benchmark macro-average accuracy among trainable PEFT baselines and matched routing controls: a 69.31% average over CMB, CMExam, MedQA, and MedMCQA. It improves over MoELoRA by 0.89 percentage points while using 28.1% fewer trainable parameters; a paired, benchmark-stratified bootstrap over final predictions gives a 95% confidence interval of [0.42, 1.37] for this macro-average gain. Basic controls show that BiRG-LoRA also improves over vanilla LoRA r16 and active-rank-matched LoRA r4 by 0.83 macro points, and an evaluation-time weak-axis perturbation check suggests that performance is not brittle to moderate tag noise. The results support a bounded claim: clinically structured rank allocation improves cross-benchmark medical QA under a matched single-seed protocol, while training-seed variance remains future work. I Introduction Medical large language models (LLMs) are commonly evaluated with exam-style multiple-choice benchmarks such as CMB, CMExam, MedQA, and MedMCQA [ 21 , 11 , 6 , 15 ] . These benchmarks mix heterogeneous medical specialties, professional roles, and reasoning operations. A question about drug mechanism, a treatment decision, a public-health policy item, and a nursing-action item may all share the same answer format, but they stress different regions of medical knowledge. This makes medical QA a difficult setting for ordinary parameter-efficient fine-tuning: a single fixed LoRA update can improve one subset while weakening another, whereas multi-adapter mixtures add storage, routing, and calibration complexity. Recent LoRA routing methods address this problem by composing or selecting adapters [ 14 , 17 , 23 , 9 ] , or by training mixture-of-LoRA experts [ 13 , 8 , 10 , 22 ] . A more fine-grained direction is rank-wise routing, where the rank channels of a single LoRA are treated as small experts rather than training many complete adapters [ 27 ] . This is attractive for medical QA because it promises sparse activation and parameter sharing. However, a purely generic rank-wise expert view misses a crucial point: in medicine, routing signals are not arbitrary task IDs. They have meaningful structure. Specialty/profession priors, clinical-operation priors, and hidden semantic cues each describe different sources of adaptation conflict. We therefore ask: can a single LoRA adapter use clinically meaningful axes to allocate its internal rank channels, while remaining competitive with multi-adapter and LoRA-MoE baselines? We propose BiRG-LoRA, a biaxial rank-gated LoRA method. BiRG-LoRA stores one rank- R adapter per target module but activates only \kappa rank atoms per question. The rank gate combines four signals: a hidden-state branch, a specialty/profession branch, a clinical-operation branch, and an interaction branch. A scalar injection coefficient controls the strength of the selected low-rank update. Figure 1 summarizes the design. This formulation makes clinical structure part of the adapter itself. Instead of assigning every question to the same fixed rank subspace, BiRG-LoRA learns which rank atoms should be used for different combinations of content domain, clinical operation, and question semantics. Under a matched Qwen3-8B protocol with 4,200 CMB training examples and 400 update steps, BiRG-LoRA reaches an average accuracy of 69.31% across CMB, CMExam, MedQA, and MedMCQA. It outperforms trainable MoELoRA by 0.89 points while using 66.59M rather than 92.60M trainable parameters. It also improves over vanilla LoRA controls, including rank-16 LoRA and an active-rank-matched rank-4 LoRA. Paired bootstrap tests over final predictions support the main macro-average gains, and evaluation-time axis perturbation suggests that the method is not brittle to moderate noise in the rule-derived clinical tags. Our contributions are: • We introduce a biaxial rank-gated LoRA formulation for medical QA, where specialty/profession and clinical-operation priors jointly guide sparse rank activation inside one adapter. • We add an input-conditioned injection coefficient that decides how strongly the selected adapter update should affect the base representation. • We provide a matched evaluation against base Qwen3-8B, vanilla LoRA r16/r4/r24, MoELoRA, DoRA, a generic rank-wise LoRA control inspired by SMoRA, and adapter-library baselines including MedAdapter, Arrow, MeteoRA, and LoRA-Mixer style routing. • We report paired, benchmark-stratified bootstrap tests and sign tests over final predictions, showing that the main macro-average gains over MoELoRA, vanilla LoRA, and a generic rank-wise control are unlikely to be explained by evaluation-set sampling noise. • We report ablations and robustness checks for clinical-only, hidden-only, no-injection, top- k , orthogonality, axis contrast, and weak-axis noise, suggesting that the full biaxial design is not reducible to generic rank-wise activation or fragile metadata lookup. Fig. 1: BiRG-LoRA uses one shared LoRA basis and dynamically selects rank atoms. The gate combines hidden semantics, a specialty/profession axis, a clinical-operation axis, and their interaction. The selected top- k diagonal mask and injection coefficient decide which rank atoms are active and how strongly the adapter modifies the frozen backbone. II Related Work Medical QA and medical LLM evaluation. PubMedQA, MedQA, MedMCQA, CMB, and CMExam provide complementary biomedical and exam-style QA settings across English and Chinese [ 7 , 6 , 15 , 21 , 11 ] . Medical LLM studies such as Med-PaLM and Med-PaLM 2 show strong progress but also emphasize calibration, uncertainty, and safety-sensitive validation [ 18 , 19 ] . Our work does not propose a new benchmark; it studies how a medical adapter should allocate low-rank capacity when source and target benchmarks differ by language, specialty mix, and reasoning operation. Parameter-efficient adaptation. LoRA freezes the base weights and learns low-rank update matrices [ 5 ] ; QLoRA makes this practical for quantized LLMs [ 1 ] ; and DoRA separates magnitude and direction updates for stronger low-rank adaptation [ 12 ] . Adaptive rank methods such as AdaLoRA, DyLoRA, and IncreLoRA allocate rank budgets across modules or deployment ranks [ 26 , 20 , 25 ] . BiRG-LoRA is different because the active rank subset is chosen per medical question, and the router is explicitly conditioned on clinical axes rather than only on weight importance or a global deployment budget. Mixtures and routers for LoRA. Sparse MoE models route tokens or examples to a small subset of experts [ 16 , 2 ] . LoRA-based MoE systems, including MoELoRA, MixLoRA, MING-MOE, and MoLE, train or combine multiple LoRA experts [ 13 , 8 , 10 , 22 ] . Other methods reuse adapter libraries or route over generated candidates, including Arrow, MedAdapter, MeteoRA, and LoRA-Mixer [ 14 , 17 , 23 , 9 ] . These systems motivate our baselines. The main difference is that BiRG-LoRA performs structured rank selection inside a single adapter rather than selecting among multiple complete adapter modules.

Read the complete text