SRAS: A Lightweight Reinforcement Learning-based Document Selector for Edge-Native RAG Pipelines
Rajiv Chaitanya Muttur · 2026
Read SRAS: A Lightweight Reinforcement Learning-based Document Selector for Edge-Native RAG Pipelines by Rajiv Chaitanya Muttur (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01785.txt.
Introduction
Retrieval-augmented generation answers questions from retrieved documents, but picking a fixed number of documents wastes computing and ignores answer quality, a real problem on small edge devices. Rajiv Chaitanya Muttur proposes SRAS, a document selector trained by reinforcement learning to choose documents with an eye on the final answer.
The learned policy is tiny, under a megabyte, trained with proximal policy optimization and a reward blending relaxed F1 with BERTScore, and it answers within a second on a plain processor. It beat supervised and random selectors on a synthetic question benchmark and generalized to real data, scoring well on SQuAD v2 without domain tuning. The work shows reinforcement learning can fit the tight budgets of on-device retrieval.
Catalog record
| CREATOR | Rajiv Chaitanya Muttur |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-01785
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SRAS: A Lightweight Reinforcement Learning-based Document Selector for Edge-Native RAG Pipelines Rajiv Chaitanya Muttur Abstract Retrieval-Augmented Generation (RAG) systems often rely on fixed top- 𝒌 k document selection mechanisms that ignore downstream generation quality and impose computational overheads. We propose SRAS (Sparse Reward-Aware Selector), a lightweight document selector trained via reinforcement learning (RL) for edge-native RAG deployment. Unlike prior RL-based retrievers that assume large memory and latency budgets, SRAS learns a compact ( ∼ \sim 0.76MB) policy using Proximal Policy Optimization (PPO), guided by a hybrid reward signal combining Relaxed F1 and BERTScore. Our method operates under tight token and compute constraints, maintaining < < 1s latency on CPU. SRAS outperforms supervised and random selectors on a synthetic QA benchmark, and generalizes to real-world data, achieving BERTScore F1 of 0.8546 on SQuAD v2 without domain-specific tuning. This work is the first to demonstrate that RL-based document selection can be made ultra-lightweight, latency-aware, and effective for on-device RAG pipelines. I Introduction Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) with external knowledge sources [ 1 , 2 ] . In a typical RAG pipeline, given a query, a retriever selects the top- k most relevant documents from a large corpus, which are then passed to a generator to produce the final output. This two-stage architecture reduces hallucinations and promotes fact-grounded responses. However, the retrieval step is crucial: If the selected documents are irrelevant or redundant, even a strong generator cannot compensate. Most existing RAG systems rely on static heuristic-based retrieval, either sparse (e.g., BM25) or dense (e.g., cosine similarity over sentence embeddings), which is not optimized for downstream task performance. These methods are agnostic to generation quality and do not incorporate feedback from the generated output, leading to a misalignment between retrieval and generation, especially in low-supervision settings. This challenge is further amplified in edge-device deployments, where constraints on compute, memory, and power necessitate compact models with low latency [ 12 ] . Large learned retrievers such as DPR [ 4 ] or ColBERT [ 5 ] are ill-suited for such environments due to their resource demands. In addition, traditional retrieval approaches do not take advantage of weak supervision signals, such as correctness of the questions and answers, to improve document selection. To address these limitations, we propose SRAS (Sparse Reward-Aware Selector), a compact neural document selector trained via reinforcement learning (RL) using sparse QA-based reward signals. SRAS replaces fixed top- k retrieval with a learned nonlinear scoring policy trained using Proximal Policy Optimization (PPO) [ 13 ] . Its hybrid reward function combines relaxed F1 and BERTScore, capturing both lexical and semantic alignment between generated and reference answers. We evaluate SRAS on a QA benchmark with synthetic supervision and compare it against strong baselines, including top- k cosine similarity, random selection, and supervised attention-based models. SRAS achieves competitive answer accuracy while maintaining a model size of less than 1MB and inference latency below 0.6s on the CPU, making it highly suitable for real-world edge deployment. Our ablation studies underscore the effectiveness of hybrid reward shaping, supervised warmup, and curriculum learning in enabling stable and sample-efficient RL training. By bridging RL with retrieval, SRAS opens a new direction for adaptive, task-aware, and lightweight RAG systems. Our main contributions are as follows: • We propose SRAS, a lightweight, reinforcement learning-based document selector that models query-document interactions and learns from sparse QA supervision. • We introduce a hybrid reward signal combining relaxed F1 and BERTScore to guide learning without requiring dense annotations. • We show that SRAS achieves strong QA performance under edge-device constraints, with sub-1MB model size and sub-1s inference latency. • We provide detailed ablations highlighting the importance of reward shaping, supervised warmup, and curriculum learning for effective policy training. • We evaluate SRAS zero-shot on 250 real-world QA examples from SQuAD v2 and demonstrate strong semantic generalization. II Related Work II-A Static and Learned Retrieval in RAG Pipelines Retrieval-Augmented Generation (RAG) architectures augment language models with external documents retrieved from a corpus [ 1 , 2 ] . Traditional approaches rely on static similarity-based retrieval using sparse methods like BM25 or dense methods such as Sentence-BERT [ 3 ] . These methods are efficient but lack adaptivity as they do not incorporate downstream task feedback or optimize retrieval for answer quality. To address this, learned retrievers such as Dense Passage Retrieval (DPR) [ 4 ] and ColBERT [ 5 ] have been proposed. These models are trained to embed queries and documents into a shared vector space using contrastive objectives. While they improve retrieval accuracy, their large size and compute requirements make them impractical for low-latency or edge scenarios. II-B Reinforcement Learning for Retrieval and Ranking Reinforcement learning (RL) has been explored in information retrieval to optimize ranking policies from feedback signals [ 6 , 7 ] . In QA systems, RL has been used to train retrievers using answer-level feedback rather than explicit document labels [ 8 , 9 ] . However, most prior work either trains full retriever encoders or combines retrieval and generation into a single RL agent, leading to high model complexity. Recent work has explored modular training using reward signals derived from generation outputs to refine retrieval [ 10 ] . Nevertheless, these methods still rely on large pretrained encoders, making them unsuitable for constrained environments. II-C Document Selection under Resource Constraints Deploying RAG systems on edge devices introduces unique challenges: memory footprint, inference latency, and power consumption must be minimized without sacrificing accuracy [ 12 ] . Most learned retrievers are ill-suited for such settings, requiring GPU acceleration and large memory buffers. Lightweight selectors using shallow architectures or projection-based scoring functions have been proposed in the context of model compression and neural ranking [ 11 ] . However, their adaptation to QA-driven RAG pipelines with sparse supervision remains under-explored. Moreover, few studies directly target RL-based document selection under edge constraints. In contrast, our work introduces a compact feedforward selector that learns to choose relevant documents using sparse QA-derived rewards. By decoupling retrieval from heavy encoders and focusing on learning from weak supervision, SRAS offers a new paradigm for low-resource RAG. III Methodology We propose a modular, reinforcement learning-based retrieval framework designed for edge-friendly RAG deployments. At the core of this framework is SRAS (Sparse Reward-Aware Selector), a compact neural selector trained to optimize document selection using weak supervision derived from downstream QA performance. This section outlines the architecture of the pipeline, the reward formulation, and the training procedure using Proximal Policy Optimization (PPO) [ 13 ] . III-A Pipeline Overview Our SRAS-enhanced RAG pipeline consists of four modular stages: 1. Corpus Preprocessing and Embedding: All raw documents are preprocessed into a flat JSON structure and embedded using a lightweight Sentence-BERT encoder (MiniLM-L6-v2) [ 3 ] , with outputs cached in .npy and .pt formats for reuse. 2. Synthetic QA Pair Generation: A generative model ( valhalla/t5-base-qg-hl ) generates synthetic question-answer pairs over the corpus. These serve as training samples for evaluating document selection effectiveness. 3. Document Selection via SRAS: Given a question embedding and a candidate set of document embeddings (typically n=8 ), SRAS assigns scores via a lightweight feedforward scoring function that models interactions between query