Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization
João Coelho, Bruno Martins, João Magalhães, Chenyan Xiong · 2025
Read Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization by João Coelho, Bruno Martins, João Magalhães, Chenyan Xiong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19307.txt.
Introduction
Fake search queries made by language models can train search engines cheaply, but poor fakes hurt results. Joao Coelho and co-authors improve those fakes in a 2025 paper. Instead of filtering bad pairs with an outside scorer, they feed ranking signals straight into the query maker.
Their method uses preference training: the maker learns to favor queries that a ranker scores as more useful. After training, query and document pairs earn higher relevance marks. Search models trained on the new data beat baselines on a standard benchmark. The authors say baking quality into generation beats cleaning up afterward. Search teams with few labeled pairs gain a stronger source of training data.
Catalog record
| CREATOR | João Coelho, Bruno Martins, João Magalhães, Chenyan Xiong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19307
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization João Coelho, Bruno Martins, João Magalhães, Chenyan Xiong Abstract . Neural retrieval models excel in Web search, but their training requires substantial amounts of labeled query-document pairs, which are costly to obtain. With the widespread availability of Web document collections like ClueWeb22, synthetic queries generated by large language models offer a scalable alternative. Still, synthetic training queries often vary in quality, which leads to suboptimal downstream retrieval performance. Existing methods typically filter out noisy query-document pairs based on signals from an external re-ranker. In contrast, we propose a framework that leverages Direct Preference Optimization (DPO) to integrate ranking signals into the query generation process, aiming to directly optimize the model towards generating high-quality queries that maximize downstream retrieval effectiveness. Experiments show higher ranker-assessed relevance between query-document pairs after DPO, leading to stronger downstream performance on the MS MARCO benchmark when compared to baseline models trained with synthetic data. Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730162 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Retrieval models and ranking † † cc-license: by 1. Introduction The effective training of neural retrieval models benefits from large-scale query-document relevance annotations ( Fang et al., 2024 ) . However, acquiring such data remains a bottleneck due to its high costs. To address this limitation, recent work has explored using Large Language Models (LLMs) to generate synthetic queries from documents ( Dai et al., 2023 ; Lee et al., 2024a ; Wang et al., 2024 ) , enabling retrieval models to be trained without full reliance on human annotations. Combined with the availability of large Web corpora such as ClueWeb22 ( Overwijk et al., 2022 ) , this establishes LLM query generators as a scalable approach for training data creation. Despite this potential, the quality of synthetic queries varies significantly, with studies showing that it still lags behind human-annotated data ( Fang et al., 2024 ) , which degrades downstream effectiveness ( Gospodinov et al., 2023 ) . To mitigate this problem, approaches for query generation consider post hoc filtering, e.g., using an external ranking signal ( Jeronymo et al., 2023 ) or through consistency checking during negative sampling ( Dai et al., 2023 ) . While this improves query quality, it does not guide the query generator toward producing queries that maximize retrieval performance. In this work, instead of post hoc query filtering, we propose a framework that directly integrates ranking feedback into the query generation process using Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) . To this end, we generate multiple queries per document, score them using a ranking model, and build a preference dataset to optimize the query generator toward producing high-quality queries. This approach enables the model to learn directly from ranking signals, aiming to improve the overall quality of queries used for dense retrieval training. More specifically, we independently consider ranking rewards from point-wise re-rankers and list-wise LLM prompting techniques, demonstrating the framework’s flexibility and opening future directions for reward selection. We conduct experiments on the MS MARCO dataset ( Nguyen et al., 2016 ) , i.e. a standard benchmark for passage and document retrieval, to evaluate the impact of DPO-optimized queries on retrieval performance. Our framework demonstrates three key contributions: (1) a training paradigm that directly aligns query generation with ranking objectives, diminishing the need for post hoc filtering inefficiencies due to increased ranker-assessed relevance; (2) flexibility in integrating diverse ranking signals, including both point-wise re-rankers and LLM-based list-wise prompting; and (3) empirical validation, showing that retrievers trained with synthetic data from our generation pipeline outperform other open-source alternatives in the Web domain. Our models and code are available in a public GitHub repository: https://github.com/cxcscmu/dpo-query-generation . 2. Related Work Transformer-based bi-encoders are the standard architecture for dense retrieval ( Xiong et al., 2021 ; Karpukhin et al., 2020 ) , typically trained with contrastive objectives and hard negative mining strategies such as ANCE ( Xiong et al., 2021 ) . Retrieval-aligned pre-training on in-domain corpora is also commonly adopted to improve retrieval effectiveness ( Gao and Callan, 2022 ; Lu et al., 2021 ; Xiao et al., 2022 ; Lee et al., 2019 ; Ma et al., 2022 ; Ma et al., 2024 ) . The community has explored generating synthetic queries from documents for model training. Approaches such as Doc2Query ( Nogueira et al., 2019 ) and DocT5Query ( Nogueira and Lin, 2019 ) train a lightweight Transformer ( Vaswani et al., 2017 ) on labeled query–document pairs to expand document representations. Subsequent work ( Gospodinov et al., 2023 ) demonstrates that filtering out hallucinated queries can further enhance downstream performance. More recent methods leverage LLMs. For instance, InPars ( Bonifacio et al., 2022 ) employs few-shot prompting with filtering based on generation probability, while InPars-v2 ( Jeronymo et al., 2023 ) leverages a supervised ranker for query filtering. Similarly, Promptagator ( Dai et al., 2023 ) uses task-specific prompts to perform few-shot query generation. Building on these approaches, the Gecko model ( Lee et al., 2024a ) iteratively refines synthetic queries through a process involving retrieval, re-ranking, positive relabeling, and hard negative sampling, thereby improving the quality of the training data. Recent work has explored reinforcement learning to improve synthetic query generation. Token-level Proximal Policy Optimization (TPPO) ( Ouyang et al., 2024 ; Schulman et al., 2017 ) has been applied to query suggestion tasks, optimizing generation based on token-level rewards derived from user interaction histories. While this setting targets interactive scenarios rather than offline generation from documents, it highlights the promise of reinforcement learning for enhancing query quality. 3. Methodology This section describes our query generation process, and details the training of retrieval models using the resulting synthetic queries. 3.1. Query Generation We start from a baseline query generator \mathcal{M}_{g} (e.g., an LLM of choice or an existing model fine-tuned for query generation). To synthesize queries, we follow a contrastive prompting strategy ( Xiong et al., 2020 ) where given a document d , a negative example d^{-} is sampled from a corpus. Then, the model is prompted to generate a query q that is specific to d , while being less relevant to d^{-} , i.e. q\sim\mathcal{M}_{g}(d,d^{-}) . For simplicity, we represent this generation process as \mathcal{M}_{g}(d) , assuming implicit contrastive prompting. By aiming to increase (q,d) similarity, this strategy mitigates retrieval training scenarios where poorly generated queries allow other corpus documents to dominate over the original positive during negative sampling, which is a source of noise. 3.2. Aligning the Query Generator Instead of filtering queries generated by \mathcal{M}_{g} , we hypothesize that the query generator can be guided towards better downstream retrieval performance. To this end, we propose leveraging a reward model to obtain a preference dataset for DPO alignment. Given the baseline generator \mathcal{M}_{g} and a corpus \mathcal{C} , we start by sampling a random pool of documents from \mathcal{C} , denoted as \mathcal{D} . For each document in \mathcal{D} , we use \mathcal{M}_{g} to