NExT-Search: Rebuilding User Feedback Ecosystem for Generative AI Search Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research

Sunhao Dai, Wenjie Wang, Liang Pang, Jun Xu, See-Kiong Ng, Ji-Rong Wen, Tat-Seng Chua · 2025

Read NExT-Search: Rebuilding User Feedback Ecosystem for Generative AI Search Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research by Sunhao Dai, Wenjie Wang, Liang Pang, Jun Xu, See-Kiong Ng, Ji-Rong Wen, Tat-Seng Chua (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14680.txt.

Introduction

Sunhao Dai and co-authors argue that AI search needs a rebuilt feedback system in this 2025 conference paper. Generative search answers questions directly, sparing users the work of browsing pages. That ease comes at a hidden cost to how search improves.

Classic web search learned from clicks and reading time on each page. AI search runs a longer chain, from splitting the query to drafting the answer. User signals blur across that chain, so models learn less from each visit. NExT-Search proposes a new ecosystem for gathering useful feedback. It aims to restore the loop that made web search steadily better. Search firms face this design problem right now.

Catalog record

CREATORSunhao Dai, Wenjie Wang, Liang Pang, Jun Xu, See-Kiong Ng, Ji-Rong Wen, Tat-Seng Chua
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14680

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NExT-Search: Rebuilding User Feedback Ecosystem for Generative AI Search Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730353 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Users and interactive retrieval Sunhao Dai, Wenjie Wang, Liang Pang, Jun Xu, See-Kiong Ng, Ji-Rong Wen, Tat-Seng Chua Abstract . Generative AI search driven by large language models (LLMs) is reshaping information retrieval by offering end-to-end answers to complex queries, reducing users’ reliance on manually browsing and summarizing multiple web pages. However, while this paradigm enhances convenience, it disrupts the feedback-driven improvement loop that has historically powered the evolution of traditional Web search. Web search can continuously improve their ranking models by collecting large-scale, fine-grained user feedback (e.g., clicks, dwell time) at the document level. In contrast, generative AI search operates through a much longer search pipeline—spanning query decomposition, document retrieval, and answer generation—yet typically receives only coarse-grained feedback on the final answer. This introduces a feedback loop disconnect , where user feedback for the final output cannot be effectively mapped back to specific system components, making it difficult to improve each intermediate stage and sustain the feedback loop. To address this limitation, we envision NExT-Search , a next-generation paradigm designed to reintroduce fine-grained, process-level feedback into generative AI search. NExT-Search integrates two complementary modes: User Debug Mode , which allows engaged users to intervene at key stages—such as refining query decomposition, rating retrieved documents, and editing initial generated responses—and Shadow User Mode , where a personalized user agent simulates user preferences and provides AI-assisted feedback for less interactive users. As these feedback signals serve as valuable resources for refining the whole search pipeline, we also introduce a feedback store mechanism that encourages users to share and monetize their debugging efforts, further incentivizing participation. Furthermore, we envision how these feedback signals can be leveraged through online adaptation , which refines current search outputs in real-time, and offline update , which aggregates interaction logs to periodically fine-tune query decomposition, retrieval, and generation models. By restoring human control over key stages of the generative AI search pipeline, we believe NExT-Search offers a promising direction for building feedback-rich AI search systems that can evolve continuously alongside human feedback. † † cc-license: by 1. Introduction Figure 1. Comparison of the paradigm of traditional web search engines and generative AI search engines. (a) Web search retrieves and ranks results, presenting them as a Search Engine Results Page (SERP), where user feedback on the document level can be directly leveraged to update the ranking model. (b) Generative AI search automates multiple steps to generate direct answers, but its extended pipeline complicates the effective use of user feedback for refining individual components. Search engines have served as a primary gateway to information access for decades, helping users address an enormous spectrum of information needs ( Manning 2009 ; Chowdhury 2010 ; Croft et al. 2010 ) . Despite continuous advances in ranking algorithms ( Burges et al. 2005 ; Xu et al. 2018 ) , user interface design ( Hearst 2009 ) , and large-scale log data analysis ( Joachims 2002 ; Craswell et al. 2008 ) , recent estimates suggest that nearly half of all Web search queries fail to yield relevant results 1 1 1 https://blogs.microsoft.com/blog/2023/02/07/reinventing-search-with-a-new-ai-powered-microsoft-bing-and-edge-your-copilot-for-the-web/ . Many of these queries lie on the complex end of the search spectrum: they require users to break down their search goal and iteratively check different returned search results, aggregating disparate pieces of information in a manual, time-consuming process ( White 2024 ; Hassan Awadallah et al. 2014 ) . Recently, large language model (LLM)–driven generative AI search systems have promised to address these more complex queries in an end-to-end fashion ( Zhu et al. 2023 ; Li et al. 2024 ; Xiong et al. 2024 ; Zhai 2024 ) . Users can now pose open-ended or creative requests (e.g., “Plan a trip to attend SIGIR 2025” ), and the generative AI search automatically parses them, retrieves relevant documents in smaller chunks, and then aggregates and synthesizes the extracted information into a cohesive answer ( Gao et al. 2023 ; Fan et al. 2024 ) . Thus, by automating many of the steps traditionally performed by users in Web search, generative AI search expands the task boundary of search, enabling users to solve complex, multi-faceted queries with reduced effort and cognitive load. However, by first revisiting and comparing the foundational paradigms of both traditional Web search and advanced generative AI search in Section 2 and Figure 1 , we find that while the paradigm of generative AI search introduces notable advantages, it also loses a critical component that has historically driven the success of traditional search: the user feedback ecosystem . • Traditional Web search thrives on a feedback-driven improvement loop , where ranking models continuously evolve by leveraging large-scale user feedback on search results. Users can provide feedback at the document level, such as clicks, dwell time, and bounce rates. These fine-grained signals serve as direct supervision for refining retrieval and ranking models, enabling search engines to iteratively enhance result relevance and improve search quality over time ( Kelly and Teevan 2003 ; Agichtein et al. 2006 ; Joachims 2002 ; Joachims et al. 2017 ) . • Generative AI search operates through a much longer pipeline that directly synthesizes answers, significantly reducing human control over the search process. Users can only provide feedback in coarse-grained forms to the final generated response, such as simple likes/dislikes or written comments ( White 2024 ) . This feedback loop disconnect prevents effective attribution of user dissatisfaction to specific pipeline components—whether query decomposition, retrieval, or answer generation—making it difficult to improve intermediate stages. Thus, while generative AI search systems such as Microsoft Bing Copilot 2 2 2 https://copilot.microsoft.com and Perplexity AI 3 3 3 https://www.perplexity.ai gained remarkable traction in their early launches, they still account for only a small share of the global search engine market 4 4 4 https://gs.statcounter.com/search-engine-market-share . Addressing these limitations is crucial for enabling generative AI search engines to achieve scalable, iterative optimization while retaining the benefits of automation. To this end, we envision a new paradigm called NExT-Search , aimed at restore—and potentially enhance—the feedback-driven ecosystem for generative AI search. The central idea is to incorporate two complementary modes of interaction: an active User Debug Mode , which enables users to engage with and refine each stage of the search pipeline, and a passive Shadow User Mode , which leverages a personalized user agent to simulate feedback when users prefer minimal involvement. In the User Debug Mode, users can examine and modify different stages of the generative search workflow—such as query decomposition, document retrieval, and answer generation—providing fine-grained feedback as needed. In contrast, the Shadow User Mode engages a personalized user agent that learns from past interactions and user profiles to provide AI-assisted feedback throughout the pipeline, thereby reducing user effort while maintaining the flow of valuable supervision signals. Together, these two modes offer a potential path toward gathering higher-quality feedback at various junctures in the

Read the complete text