Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation

Evgenii Palnikov, Elizaveta Gavrilova, [6pt] HSE University · 2026

Read Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation by Evgenii Palnikov, Elizaveta Gavrilova, [6pt] HSE University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28222.txt.

Introduction

Evgenii Palnikov and Elizaveta Gavrilova weigh quality, speed, and cost for a docs-grounded helper tuned with LoRA. On a checked set of over five thousand Kubernetes question-answer pairs, adapters touching only query and value parts topped the trade-off frontier. The edge looks built-in, not just parametric, while model size sets the working zone.

Catalog record

CREATOREvgenii Palnikov, Elizaveta Gavrilova, [6pt] HSE University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE130454
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28222

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation Evgenii Palnikov, Elizaveta Gavrilova, [6pt] HSE University Abstract We study quality–latency–resource trade-offs in a documentation-grounded retrieval-augmented generation (RAG) system that uses Low-Rank Adaptation (LoRA) of the generator. We build a manually verified benchmark of 5,144 question–answer pairs over the official Kubernetes documentation and combine it with a fixed hybrid-retrieval pipeline (BGE-M3 dense, BGE-M3 native sparse, Reciprocal Rank Fusion, cross-encoder reranking). Over this benchmark we ablate 20 LoRA configurations on Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct across rank and target-module choices, and evaluate each on token-level F1, LLM-judged groundedness and correctness (pass@4), inference latency, inference memory, and training cost, all reported with bootstrap 95% confidence intervals. Pareto analysis shows that LoRA adapters acting only on the q and v attention projections consistently dominate the front, while the 3B/8B choice mainly defines operating regime. A param-matched control comparison further indicates that the q/v advantage is structural rather than purely parametric. The benchmark, selected adapters, and code are available at https://github.com/EugPal/rag-lora-tradeoffs . 1 Introduction Retrieval-augmented generation (RAG) has become a standard way to ground large language models (LLMs) in external corpora when their parametric memory is insufficient or stale ( Lewis et al. 2020 ; Gao et al. 2023 ; Wu et al. 2024 ) . For documentation-oriented assistants – where users expect answers backed by specific configuration flags, API versions, or feature gates – the bar is even higher: a deployable system must combine factual accuracy and visible support from the retrieved context with acceptable latency, inference memory, and training cost. In practice these objectives are in direct tension. Larger generators improve answer quality but increase inference memory and latency; richer retrieval and reranking add overhead; more expressive LoRA ( Hu et al. 2022 ) adapters change the cost of training. Picking a working point therefore requires reasoning about a multi-objective trade-off rather than a single quality metric. Existing work largely studies these dimensions in isolation. RAG surveys focus on retrieval/fusion/reranking design choices ( Gao et al. 2023 ; Yu et al. 2024 ) ; parameter-efficient fine-tuning surveys focus on the LoRA family in general purpose tasks ( Han et al. 2024 ; Wang et al. 2025 ) ; RAG-evaluation work focuses on quality and groundedness on standard QA sets ( Es et al. 2024 ; Yu et al. 2024 ) . A few recent papers combine RAG with LoRA, but they typically treat LoRA as a single technique – a single rank, a single target-module set – and compare it against alternatives like DoRA or full fine-tuning ( Tahir et al. 2024 ; Zhao et al. 2024a ; Baqar & Khanda 2025 ) . This leaves an open question: how does the choice of LoRA configuration – rank, target modules, base-model size – interact with retrieval quality, latency, and training cost when the rest of the RAG pipeline is held fixed? We address this question by constructing a documentation-grounded RAG benchmark and analysing a structured space of LoRA configurations on it. Our benchmark consists of 5{,}144 manually verified question–answer pairs over the official Kubernetes documentation, paired with a fixed hybrid retrieval pipeline (BGE-M3 ( Chen et al. 2024 ) dense + BGE-M3 native sparse + Reciprocal Rank Fusion ( Cormack et al. 2009 ) + a cross-encoder reranker). Over this pipeline we ablate 20 LoRA configurations on Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct (5 ranks \times 2 target-module sets per base model) against the corresponding non-adapted baselines. Each configuration is evaluated on (i) token-level F1, (ii) LLM-as-a-judge groundedness and correctness (pass@4), (iii) inference latency, (iv) inference VRAM, and (v) training time and VRAM, with bootstrap 95% confidence intervals on all point estimates and paired bootstrap on key \Delta F_{1} comparisons. To probe robustness, we re-run the full grid under 10 retrieval/prompting ablation regimes. Three findings emerge. First, on every Pareto front we examine – F1-vs-latency, F1-vs-inference-memory, and F1-vs-training-cost – non-dominated points are dominated by LoRA adapters that act only on the q and v attention projections. full_attention adapters ( q,k,v,o ) never appear on the training Pareto front and win at most 2/10 ablation regimes on groundedness. Second, the 3B/8B choice mainly determines the operating regime rather than the achievable quality: the strongest LoRA-adapted 3B configuration is statistically indistinguishable from the unadapted 8B baseline on F_{1} ( \Delta F_{1}\in[-0.021,+0.026] ), while costing roughly 9 GB less inference VRAM. Third, a param-matched control comparison – in which the total number of trainable LoRA parameters is held constant across the q/v and full-attention schemes – shows that the q/v advantage is structural and not a consequence of having more parameters. Our contributions are: (i) a manually verified, license-clean RAG benchmark over Kubernetes documentation, designed for multi-criteria evaluation; (ii) a systematic LoRA-configuration study within a fixed hybrid-retrieval pipeline, with statistical inference on all reported trade-offs; (iii) a param-matched control comparison that isolates the structural effect of the target-module choice from the effect of parameter count; (iv) the released benchmark and selected adapters, intended as a reproducible starting point for follow-up work on retrieval-augmented fine-tuning of documentation assistants. 2 Related Work RAG and retrieval components. Retrieval-augmented generation ( Lewis et al. 2020 ) has been thoroughly surveyed in recent years ( Gao et al. 2023 ; Wu et al. 2024 ) : modern RAG is not a single architecture but a family of designs that differ in how retrieval, fusion, reranking, and context use are organised. On the retrieval side, dense passage retrieval was canonicalised by Karpukhin et al. 2020 , classical sparse retrieval by BM25 ( Robertson & Walker 1994 ; Robertson et al. 1995 ; Robertson & Zaragoza 2009 ) , and training-aware extensions such as RocketQA ( Qu et al. 2021 ) and ColBERTv2 ( Santhanam et al. 2022 ) show that retrieval quality depends as much on the training scheme as on the index type. Recent multi-functional embedders – in particular BGE-M3 ( Chen et al. 2024 ) – are widely adopted in hybrid pipelines combined with cross-encoder reranking ( Nogueira & Cho 2019 ; Yoon et al. 2024 ) and rank-level fusion ( Cormack et al. 2009 ) . For documentation-grounded assistants the overall quality is therefore a property of the whole pipeline, not of the generator alone. Parameter-efficient fine-tuning. Full fine-tuning of instruction-tuned LLMs is expensive in both compute and memory; PEFT surveys ( Han et al. 2024 ; Wang et al. 2025 ) document a large literature of methods that change model behaviour while keeping the backbone frozen. Among these, LoRA ( Hu et al. 2022 ) introduces low-rank adapters on linear projections and has spawned a family of follow-ups, including QLoRA ( Dettmers et al. 2023 ) and LoRA+ ( Hayou et al. 2024 ) , plus throughput-focused training infrastructure such as ASPEN ( Ye et al. 2023 ) . Collectively, this line shows that LoRA can deliver domain adaptation without full fine-tuning. It does not, however, answer the question of which LoRA configuration – rank, target modules, base-model size – to pick when the adapter is plugged into a retrieval-augmented system. RAG evaluation, groundedness, and quality–cost trade-offs. The evaluation literature ( Yu et al. 2024 ; Es et al. 2024 ) stresses that RAG quality must be measured not only

Read the complete text