In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective

Mingchen Li, Jiatan Huang, Chuxu Zhang, Liang Zhao, Hong Yu · 2026

Read In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective by Mingchen Li, Jiatan Huang, Chuxu Zhang, Liang Zhao, Hong Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26356.txt.

Introduction

Mingchen Li and four colleagues treat retrieval-augmented generation as in-context optimization. They show one linear self-attention layer takes a gradient-descent step on a unified linearized retrieval goal. The match holds under controlled linear cases but turns data-dependent under nonlinear designs. Their method for frozen retrieval models predicts a context-tuned update to the interface, beating a baseline across seven benchmarks, in a 2026 paper.

Catalog record

CREATORMingchen Li, Jiatan Huang, Chuxu Zhang, Liang Zhao, Hong Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67875
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26356

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective Mingchen Li, Jiatan Huang, Chuxu Zhang, Liang Zhao, Hong Yu Abstract In-context learning has recently been linked to implicit gradient descent in linear self-attention models, suggesting that context can induce a forward-pass update. Retrieval-augmented generation (RAG) also relies on context, but retrieved documents are usually treated as static evidence rather than signals for adaptation. We study RAG as an in-context optimization process. First, we show that one linear self-attention layer can implement one gradient-descent step on a unified linearized RAG objective covering both projection-based and dot-product retrieval interfaces. This gives an exact regime where retrieval-augmented prediction and in-context optimization coincide. We use this result not as a literal model of LLM computation, but as a guide for adapting the interaction between queries and retrieved evidence. We then test the boundary of this correspondence: it remains stable under controlled linear extensions, but becomes feature-distribution dependent under nonlinear architectures. Finally, we turn this view into a lightweight method for frozen RAG LLMs. The method keeps the retriever and backbone fixed, and predicts a context-conditioned update to a generator-side evidence-use interface. Across seven QA benchmarks, two retrievers, and two frozen LLM backbones, this forward-only update improves a shared-interface baseline, transfers to held-out tasks, and approaches test-time gradient adaptation at much lower per-query cost. 1 Introduction Large language models (LLMs) have achieved strong performance across many natural-language tasks, but adapting them to knowledge outside their static pretraining corpus remains difficult. Retrieval-augmented generation (RAG) ( 23 ) addresses this limitation by conditioning a frozen LLM on documents retrieved from an external corpus. However, retrieval alone does not solve the full adaptation problem. After relevant documents are retrieved, the model must still decide how to use them for a new task, domain, or query distribution. Existing RAG systems usually address this problem in one of three ways. The first is to keep both the retriever and generator fixed, and simply prepend retrieved documents to the input. This strategy is efficient, but it treats retrieved documents as static evidence and gives the model no mechanism to adjust how evidence should be used. The second is to fine-tune the retriever, the generator, or both. This can improve task performance, but it requires additional training and can be expensive when the task or domain changes. The third is to use in-context learning (ICL), where a few input-output examples are provided at inference time. ICL is attractive because it avoids full model retraining, but it is still unclear whether these examples merely provide extra demonstrations, or whether they can induce a more systematic update to how a RAG model uses retrieved evidence. This paper asks a simple question: can retrieved evidence and a few RAG examples act not only as context to read from, but also as a signal for adapting how the model uses evidence? Answering this question requires connecting two views that have mostly been studied separately. On one side, recent theory shows that, under linear self-attention, in-context learning can implement gradient descent on the examples in the context ( 41 ; 2 ; 25 ) . This suggests that context can behave like a forward-pass update, rather than only as additional input text. On the other side, RAG introduces structure that is absent from standard ICL theory: a query, retrieved documents, query-evidence interactions, and a generator that must combine them to produce an answer. It remains unclear whether the gradient-descent view of ICL extends to retrieval-augmented prediction, and whether such a view can guide practical adaptation in real RAG systems. We study RAG from this in-context optimization perspective. Our goal is not to claim that modern retrieval-augmented LLMs literally perform gradient descent during inference. Instead, we use a controlled linear setting to identify where such an update would act. In linear RAG, the relevant update acts on the interaction between the query and retrieved evidence. This gives a simple design principle for LLM-scale RAG: rather than changing which documents are retrieved, we adapt how the frozen generator uses the retrieved documents. Guided by this principle, we propose a forward-only adaptation method for frozen RAG LLMs. The method keeps both the external retriever and the LLM backbone fixed, and adapts only a lightweight generator-side evidence-use interface implemented with LoRA. At inference time, given new few-shot RAG demonstrations, the predictor produces the update in a single forward pass, enabling the generator to adjust how it uses retrieved documents without re-training on the new dataset. We develop this idea in three steps. First , we prove that one linear self-attention layer can implement one gradient-descent step on a unified linearized RAG objective covering both projection-based and dot-product retrieval interfaces. Second , we test how far this correspondence extends beyond the exact construction. A trained self-attention layer closely matches the constructed gradient-descent predictor under controlled linear shifts, varying document counts, and stacked depths, while nonlinear architectures and real-world regression data reveal a clear dependence on feature distribution. Third , we use the optimization view to guide LLM-scale RAG adaptation. Across seven QA benchmarks, two retrievers, and two frozen LLM backbones, the predicted update improves a shared-interface baseline, transfers to held-out tasks, and approaches test-time gradient adaptation at much lower per-query cost. Our contributions are summarized as follows: • An in-context optimization view of linear RAG. We extend the ICL-as-gradient-descent perspective from generator-only prediction to retrieval-augmented prediction. We prove that one linear self-attention layer can implement one gradient-descent step on a unified linearized RAG loss covering both linear-projection and dot-product retrieval interfaces. We also show that stacking K linear self-attention layers gives a multi-step view of in-context optimization for linear RAG. • A boundary analysis beyond the exact linear setting. We test when the linear construction remains predictive and when it breaks. On synthetic linear regression tasks, a trained self-attention layer closely matches the constructed gradient-descent predictor under distribution shift, varying document counts, and stacked depths. On nonlinear architectures and four real-world regression datasets, the alignment degrades in a structured way and becomes sensitive to feature distribution. • Adapting evidence use without test-time backpropagation. We use the optimization view to guide adaptation in frozen RAG LLMs. Rather than changing the external retriever, we adapt a generator-side evidence-use interface implemented with Q/K/V LoRA modules. A small context-conditioned predictor amortizes the autograd-defined K -step update to this interface. Across seven QA benchmarks, two backbones, and two retrievers, the predicted update improves a shared-interface baseline, transfers to held-out domains, and approaches test-time gradient adaptation at much lower per-query cost. 2 Related Work Retrieval-augmented generation (RAG) conditions a language model on documents retrieved from an external corpus ( 23 ; 12 ; 19 ; 16 ; 6 ; 45 ) . Prior work has improved RAG through better retrieval, prompting, evidence fusion, and joint retriever-generator training ( 33 ; 3 ; 15 ; 46 ) . Our focus is complementary: rather than changing which documents are retrieved, we study how a frozen generator can adapt its use of already-retrieved evidence. We position this contribution relative to three lines of work. In-context learning as gradient descent. A growing line of theory interprets in-context learning as implicit optimization. Under linear self-attention, a single ICL forward pass can implement one gradient-descent

Read the complete text