DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning
Pengcheng Jiang ∗ , Jiacheng Lin , Lang Cao, Runchu Tian, SeongKu Kang † , Zifeng Wang, Jimeng Sun, and Jiawei Han University of Illinois Urbana-Champaign † Korea University · 2025
Read DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning by Pengcheng Jiang ∗ , Jiacheng Lin , Lang Cao, Runchu Tian, SeongKu Kang † , Zifeng Wang, Jimeng Sun, and Jiawei Han University of Illinois Urbana-Champaign † Korea University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00223.txt.
Introduction
Pengcheng Jiang and co-authors train language models to rewrite search queries by trial and error in a paper from 2025. Search engines live or die by the query they receive. Past model-based rewriters needed costly labeled data or a teacher model to copy.
DeepRetrieval uses reinforcement learning with retrieval scores as the reward. No reference queries are needed at all. The model learns phrasing that real search engines reward. On literature search, recall hit about sixty-five percent for papers, far above the prior best near twenty-five. Trial search recall also doubled the old mark. Learning from the engine's own feedback may beat learning from human examples.
Catalog record
| CREATOR | Pengcheng Jiang ∗ , Jiacheng Lin , Lang Cao, Runchu Tian, SeongKu Kang † , Zifeng Wang, Jimeng Sun, and Jiawei Han University of Illinois Urbana-Champaign † Korea University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00223
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning Pengcheng Jiang ∗ , Jiacheng Lin , Lang Cao, Runchu Tian, SeongKu Kang † , Zifeng Wang, Jimeng Sun, and Jiawei Han University of Illinois Urbana-Champaign † Korea University Abstract Information retrieval systems are crucial for enabling effective access to large document collections. Recent approaches have leveraged Large Language Models (LLMs) to enhance retrieval performance through query augmentation, but often rely on expensive supervised learning or distillation techniques that require significant computational resources and hand-labeled data. We introduce DeepRetrieval, a reinforcement learning (RL) approach that trains LLMs for query generation 1 1 1 In this study, we use the term “query generation” to cover both “query augmentation/rewriting” for text retrieval and “SQL query generation” for database search. through trial and error without supervised data (reference query). Using retrieval metrics as rewards, our system generates queries that maximize retrieval performance. DeepRetrieval outperforms leading methods on literature search with 65.07% (vs. previous SOTA 24.68%) recall for publication search and 63.18% (vs. previous SOTA 32.11%) recall for trial search using real-world search engines. DeepRetrieval also dominates in evidence-seeking retrieval, classic information retrieval and SQL database search. With only 3B parameters, it outperforms industry-leading models like GPT-4o and Claude-3.5-Sonnet on those tasks. These results demonstrate that our RL approach offers a more efficient and effective paradigm for information retrieval. Our data and code are available at: https://github.com/pat-jj/DeepRetrieval . (a) (b) Figure 1 : Performance Overview of DeepRetrieval. (a) Literature Search Performance (Recall@3K) on Real-World Search Engines (PubMed and ClinicalTrials.gov); (b) Performance on Classic Information Retrieval and SQL database search Tasks. 1 Introduction Information retrieval (IR) systems play a critical role in helping users find relevant information within large document collections. In today’s data-rich environment, the effectiveness of these systems directly impacts our ability to access and leverage knowledge across domains ranging from scientific literature and clinical evidence to general web content and structured databases ( Baeza-Yates et al. 1999 ; Singhal et al. 2001 ) . Traditional approaches to IR rely heavily on keyword matching and statistical methods, which often struggle with understanding the semantic meaning behind user queries ( Krovetz & Croft 1992 ; Schütze et al. 2008 ; Robertson et al. 2009 ) . The semantic gap between user information needs and their query formulations has long been recognized as a fundamental challenge in IR ( Furnas et al. 1987 ; Croft et al. 2010 ) . A user’s initial query is frequently an imperfect representation of their information need, limited by vocabulary mismatches, domain knowledge gaps, or an inability to articulate complex requirements ( Belkin et al. 1982 ; Ingwersen 1996 ; Taylor 1968 ) . Recent advances in Large Language Models (LLMs) have shown promising results in addressing these limitations through query augmentation ( Bonifacio et al. 2022 ; Mao et al. 2020 ; Nogueira & Cho 2019 ) , where LLMs expand or reformulate user queries to better capture relevant documents. By bridging the semantic gap between users’ information needs and document collections, query augmentation helps retrieve more relevant content that might otherwise be missed using traditional keyword matching ( Dai & Callan 2019 ; Yates et al. 2021 ; Gao et al. 2021 ) . However, current LLM-based approaches to query augmentation typically employ supervised learning or distillation techniques, which face several significant limitations. They require expensive computational resources to generate training data, often costing thousands of dollars for instruction tuning or distillation from larger models ( Wang et al. 2022c ; Zhou et al. 2023 ; Wang et al. 2025 ) . The quality of augmented queries depends heavily on the quality and coverage of supervision data ( Khattab et al. 2021 ) , and they rely on larger models to generate data for smaller ones, potentially introducing biases and limitations ( Beyer et al. 2022 ) . Furthermore, these approaches typically optimize for mimicking human-generated or teacher model-generated queries rather than for end-task performance ( Kudugunta et al. 2021 ; Asai et al. 2023 ) . In this work, we introduce DeepRetrieval, a novel approach that uses reinforcement learning (RL) to train LLMs for query generation/rewriting. Rather than relying on supervision data, DeepRetrieval allows the model to learn through direct trial and error, using retrieval metrics (such as NDCG ( Järvelin & Kekäläinen 2002 ) ) as reward. Our approach is inspired by recent advances in using RL to enhance reasoning capabilities in LLMs ( Guo et al. 2025 ; Lightman et al. 2023 ; Hsieh et al. 2023 ) , but specifically adapted for the IR domain. We evaluate DeepRetrieval across a diverse range of retrieval tasks, including literature search using real-world search engines (PubMed and ClinicalTrials.gov), evidence-seeking retrieval across common question-answering datasets ( Kwiatkowski et al. 2019 ; Joshi et al. 2017 ; Rajpurkar et al. 2016 ) , classic information retrieval benchmarks with BM25 ( Robertson et al. 2009 ) and dense retrieval systems ( Karpukhin et al. 2020 ) , and SQL database search for structured data access ( Yu et al. 2018 ) . DeepRetrieval significantly outperforms state-of-the-art models on literature search. It achieves 65.07% recall (vs. previous SOTA of 24.68%) for publication search and 63.18% (vs. previous SOTA of 32.11%) for clinical trial search ( Wang et al. 2025 ) . Also, it demonstrates superior performance on evidence-seeking retrieval ( Ma et al. 2021 ; Yates et al. 2021 ) , classic IR tasks ( Thakur et al. 2021 ) , and SQL database search ( Gao et al. 2023a ) , outperforming even industry-leading models like GPT-4o ( Hurst et al. 2024 ) and Claude-3.5-Sonnet ( Anthropic 2024 ) with only 3B parameters. The key contributions of our work include: [1] A novel reinforcement learning framework for query generation that directly optimizes for retrieval performance, which is a more cost-efficient paradigm for information retrieval that eliminates the need for expensive supervision data for query generation [2] A structured generation method that enables effective reasoning before query formulation. We conduct comprehensive evaluation across diverse retrieval tasks demonstrating consistent superiority over existing methods [3] We demonstrate that RL offers a more efficient and effective paradigm for enhancing IR systems, potentially changing how we approach the fundamental challenge of connecting users with relevant information across different domains and retrieval settings Figure 2 : Abstractive Overview of DeepRetrieval : Based on an input user query, the LLM generates an augmented query, which is used to retrieve documents. Format reward and retrieval reward are both computed as the feedback to update the model. 2 DeepRetrieval DeepRetrieval is an RL-based framework for training LLMs to enhance information retrieval through query generation/rewriting. Unlike existing approaches ( Lin et al. 2020 ; Ye et al. 2023 ; Ma et al. 2023 ; Wang et al. 2024b ) that rely on supervised learning with labeled query-augmentation pairs, our approach learns directly from retrieval outcomes. Figure 2 illustrates the overall architecture of our system. 2.1 Problem Formulation Let D be a collection of contexts and q be a user query. The goal of an information retrieval system is to return a set of contexts D_{q}\subset D that are relevant to q . In query augmentation,