How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809667 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Evaluation of retrieval results CCS: Information systems Web crawling CCS: Information systems Page and site ranking CCS: Information systems Language models
Riley Grossman, Songjiang Liu, Michael K. Chen, Mike Smith, Cristian Borcea, Yi Chen · 2026
Read How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809667 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Evaluation of retrieval results CCS: Information systems Web crawling CCS: Information systems Page and site ranking CCS: Information systems Language models by Riley Grossman, Songjiang Liu, Michael K. Chen, Mike Smith, Cristian Borcea, Yi Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27790.txt.
Introduction
Grossman and five colleagues study how generative AI disrupts web search, comparing Google, AI Overviews, and Gemini Flash 2.5 on 11,500 queries. AI Overviews appear for 51.5 percent of typical queries, displayed above plain results, with touchy questions often raising one. Found sources differ sharply across engines: classic search favors popular government and school sites, while AI engines favor Google-owned content. Sites blocking Google's AI crawler appear less often in AI Overviews, which prove less steady across repeat runs and less robust to small query edits.
Catalog record
| CREATOR | Riley Grossman, Songjiang Liu, Michael K. Chen, Mike Smith, Cristian Borcea, Yi Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55855 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27790
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809667 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Evaluation of retrieval results CCS: Information systems Web crawling CCS: Information systems Page and site ranking CCS: Information systems Language models Riley Grossman, Songjiang Liu, Michael K. Chen, Mike Smith, Cristian Borcea, Yi Chen Abstract . Generative AI is being increasingly integrated into web search for the convenience it provides users. In this work, we aim to understand how generative AI disrupts web search by retrieving and presenting the information and sources differently from traditional search engines. We introduce a public benchmark dataset of 11,500 user queries to support our study and future research of generative search. We compare the search results returned by Google’s search engine, the accompanying AI Overview (AIO), and Gemini Flash 2.5 for each query. We have made several key findings. First, we find that for 51.5% of representative, real-user queries, AIOs are generated, and are displayed above the organic search results. Controversial questions frequently result in an AIO. Second, we show that the retrieved sources are substantially different for each search engine (<0.2 average Jaccard similarity). Traditional Google search is significantly more likely to retrieve information from popular or institutional websites in government or education, while generative search engines are significantly more likely to retrieve Google-owned content. Third, we observe that websites that block Google’s AI crawler are significantly less likely to be retrieved by AIOs, despite having access to the content. Finally, AIOs are less consistent when processing two runs of the same query, and are less robust to minor query edits. Our findings have important implications for understanding how generative search impacts website visibility, the effectiveness of generative engine optimization techniques, and the information users receive. We call for revenue frameworks to foster a sustainable and mutually beneficial ecosystem for publishers and generative search providers. † † cc-license: by 1. Introduction With the rapid advancement of large language models (LLMs), users have begun to use AI chatbots (e.g., ChatGPT) as a replacement for traditional search engines ( Singh 2025 ; Bailyn 2025 ; Venditti 2025 ) . In response, traditional search engines have integrated LLM capabilities into their search functionalities, thereby affecting traditional search engine users in a way that AI chatbots do not. For example, while users must choose to use AI chatbots, Google places its AI Overview (AIO) above the traditional search engine results pages (SERP) by default (see Figure 1 ). 1 1 1 Users can temporarily remove AIOs by adding “-AI” to a query or scrolling to the “Web” tab in the search results. We use the term generative search engines to refer to any AI-powered systems that retrieve relevant sources in response to a user query and generate a summary of the content in those sources. Generative search engines offer convenience to users ( Liang et al. 2025 ; Stadler et al. 2024 ; Dai et al. 2025a ) and are rapidly gaining popularity. However, it remains unclear how the sources retrieved by generative search engines compare with those returned by traditional search, what implications such differences have for users, and how these differences affect websites. Users rely on the retrieved sources to obtain information, particularly those ranked highly by search engines ( Baye et al. 2016 ; Sharma et al. 2019 ; Zilincan 2015 ) . Thus, it is critical that search engines consistently retrieve quality sources (e.g., for supporting an informed electorate in democratic society ( Dai et al. 2024a ; Bozdag and Van Den Hoven 2015 ; Steiner et al. 2022 ) ). Furthermore, for usability, it is important that search engines retrieve consistent results in the presence of minor changes to the query syntax that preserve intent ( Bailey et al. 2017 ; Hannak et al. 2013 ; Jiang et al. 2015 ; Batri et al. 2025 ) . Figure 1 . Google’s AI Overview Interface From a website’s perspective, there is growing concern that generative search reduces website traffic, as users increasingly obtain information directly from generated summaries rather than from sources listed in traditional SERPs ( Bhattacharya 2025 ) . Historically, websites have relied on Search Engine Optimization (SEO) services to improve their rankings in traditional search results in order to increase traffic, thereby increase advertising revenue or sales opportunities. More recently, websites have begun exploring strategies to offset declining traffic by increasing their presence as cited sources in generative search results ( McDonald 2025 ; Hohli 2025 ) . Although some companies offer Generative Engine Optimization (GEO) services to increase website visibility in generative search results, with some evidence of its potential ( Aggarwal et al. 2024 ; Pfrommer et al. 2024 ; Chen et al. 2025 ) , the effectiveness of GEO is contested ( Puerto et al. 2025 ) . More fundamentally, there is a lack of understanding of how the characteristics of sources cited by generative search differ from those retrieved by traditional search engines. In this paper, we conduct a large-scale comparative analysis of the sources retrieved by traditional and generative search engines. We construct a benchmark query set of 11,500 queries, along with two additional time-sensitive query sets, spanning diverse topics, user intents, and syntactic variations. Using SerpAPI 2 2 2 https://github.com/serpapi and the Gemini API, we collect ranked lists of retrieved sources from Google Search (i.e., traditional SERP), AIOs, and Gemini 2.5 Flash. We then use Jaccard similarity and rank-biased overlap (RBO) to quantify differences between the sources retrieved by each engine, and also to evaluate each system’s internal consistency across repeated runs and in response to variations in query syntax, user location, and device. We also collect domain-level characteristics (e.g., popularity and content category) of the retrieved sources to assess the source preferences of each search engine. We summarize our findings as follows: • AIOs have a substantial presence (e.g., 51.5% of representative real-user queries), particularly in response to long, informational queries formatted as a question. • Traditional SERP, Gemini, and AIO exhibit low average similarity in terms of the retrieved sources for each query (i.e., Jaccard similarities between 0.11 and 0.18). • Generative search engines are significantly less likely to retrieve sources from popular websites, educational or governmental institutions, and websites that block Google’s AI bot (even though AIOs can still access the content). Generative search engines retrieve significantly more content from Google-owned websites. • Generative search is less consistent across runs of the same query, and less robust to changes in device type or query syntax, compared to traditional search. • Although AIOs are rarely generated for trending queries (8.1%), they are prevalent for sensitive queries (e.g., 93.8% of political queries). Despite the importance of these queries to maintaining an informed electorate, generative search engines rely on less credible sources. They also often take a stance in the generated text summary (33.4% of AIO, and 5.6% of Gemini, responses, respectively). Our findings have important societal