Demographically-Inspired Query Variants Using an LLM Conference: Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR); July 18, 2025; Padua, Italy Proceedings of the 2025 International ACM SIGIR Conference on Innovative Conce
Marwah Alaofi, Nicola Ferro, Paul Thomas, Falk Scholer, Mark Sanderson · 2025
Read Demographically-Inspired Query Variants Using an LLM Conference: Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR); July 18, 2025; Padua, Italy Proceedings of the 2025 International ACM SIGIR Conference on Innovative Conce by Marwah Alaofi, Nicola Ferro, Paul Thomas, Falk Scholer, Mark Sanderson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17644.txt.
Introduction
Marwah Alaofi and co-authors use a large language model to create search query variants that reflect the diversity of real search users. Their 2025 paper starts from an old vision of an ideal test collection for information retrieval. Variants keep the meaning of an original query but phrase it as users with different language skill or topic knowledge might.
Tests confirmed the variants matched the user profiles they aimed at. When retrieval systems were ranked with the variants, the order changed, and some profiles got much better service than others. The authors offer the method as a new lens for system evaluation, one that shows both how rankings shift and how service varies across users. The catalog title carries stray proceedings text, so the head title governs.
Catalog record
| CREATOR | Marwah Alaofi, Nicola Ferro, Paul Thomas, Falk Scholer, Mark Sanderson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17644
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Demographically-Inspired Query Variants Using an LLM Conference: Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR); July 18, 2025; Padua, Italy Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) (ICTIR ’25), July 18, 2025, Padua, Italy DOI: 10.1145/3731120.3744608 ISBN: 979-8-4007-1861-8/2025/07 CCS: Information systems CCS: Information systems Query representation CCS: Information systems Retrieval effectiveness Marwah Alaofi, Nicola Ferro, Paul Thomas, Falk Scholer, Mark Sanderson Abstract . This study proposes a method to diversify queries in existing test collections to reflect some of the diversity of search engine users, aligning with an earlier vision of an ‘ideal’ test collection. A LLM ( LLM ) is used to create query variants: alternative queries that have the same meaning as the original. These variants represent user profiles characterised by different properties, such as language and domain proficiency, which are known in the IR ( IR ) literature to influence query formulation. The LLM ’s ability to generate query variants that align with user profiles is empirically validated, and the variants’ utility is further explored for IR system evaluation. Results demonstrate that the variants impact how systems are ranked and show that user profiles experience significantly different levels of system effectiveness. This method enables an alternative perspective on system evaluation where we can observe both the impact of user profiles on system rankings and how system performance varies across users. † † cc-license: by 1. Introduction The ‘ideal’ test collections envisioned by Spärck-Jones and Van Rijsbergen ( Spärck-Jones and van Rijsbergen 1975 ) called for diversity in both documents and ‘requests’ (now known as queries) by varying content, type, source, and origin. This vision reflects Cleverdon 1961 ’s early emphasis on including ‘representative’ questions in evaluation. Although the creation of test collections – such as those developed in TREC – was largely inspired by this vision, it has not been fully realised. Most focus has been on document pooling and consistency of relevance judgments; far less attention has been given to the diversity of queries. Past work has shown that factors like modality ( Church et al. 2007 ; Harvey and Pointon 2017 ) , domain expertise ( Monchaux et al. 2015 ; White et al. 2009 ) , age ( Duarte Torres et al. 2010 ; Bilal and Gwizdka 2018 ) , and language proficiency ( Chu et al. 2015 ) influence query formulation, and consequently affect the accuracy of search results and user satisfaction. However, both capturing this variation and understanding its utility in test collection remain limited. Most system evaluation is based on so-called generic queries; how systems perform in other contexts, e.g., mobile users, or those with limited domain expertise, remains under-explored. This raises the question: how would the effectiveness of IR systems differ under a broader spectrum of queries? This calls for innovative approaches to develop cost-effective, comprehensive evaluation environments. This work seeks to realise the original vision of the ‘ideal’ test collections by reintroducing the often-overlooked element of user diversity – explicitly controlling for variables that influence query formulation. The work describes an approach to expand a one query representation of information needs to better represent a diverse range of search engine users. It introduces a simulation of query variants : alternative queries that have the same meaning as an original (seed) query. The simulated incorporation of user properties is achieved through three LLM -based transformation methods applied to seed queries. These methods integrate profiles for specific personas (e.g., Anne the retired nurse), user groups (e.g., Voice search users), and single textual transformations (e.g., paraphrasing ). Table 1 shows examples of variants from the three methods, along with their effectiveness scores on two IR systems. The work aims to validate LLM -generated query variants and explore their retrieval impact . A variant is deemed valid if it is formulated differently yet shares the same meaning as the seed query, and if it reasonably manifests its profile. The study explores the impact of variants on the effectiveness of IR systems, and offers new insights regarding their utility in system evaluation. An overview of the experimental design is given in Figure 1 . We explore the following research questions: RQ1 Are LLM -generated query variants valid? 1.1 Lexical Variation: Do variants differ lexically from seed queries? 1.2 Semantic Similarity: Are variants semantically similar to seed queries? 1.3 Profile Alignment: Are variants likely to be generated by their respective profiles? RQ2 Do the generated query variants change the outcome of system evaluation? 2.1 System Ranking: How do variants impact on relative system effectiveness rankings? 2.2 User Equity: Do systems serve users equally? (E.g., is a mobile user served as effectively as a voice search user?) Figure 1 . An overview of the experiment pipeline and the research questions. We make the following contributions in understanding the utility of an LLM in simulating queries and creating relevance labels: 1 1 1 Data is available at: https://github.com/MarwahAlaofi/demo-qv C1 An adaptable evaluation pipeline that, based on search engine user properties, generates query variants for fine-grained system evaluations. This assists in highlighting problems faced by certain demographics, and identifying where the system performs effectively or inadequately. C2 A publicly available dataset of approximately 7K query variants for the DL21 ( DL21 ) and DL22 ( DL22 ), with human assessments covering a 10% sample of each profile, to support studying similarity and profile alignment. C3 An evaluation of GPT-4o prompted relevance labels for DL21 and DL22 . The labels – 146K query-passage pairs – are made available publicly, with 13K labels human-judged by the NIST ( NIST ), enabling further research into LLM relevance labels. Table 1. An example seed query from DL22 and its LLM-generated query variants and their NDCG@10 scores using two IR systems. Profile Variant BM25 ANCE Seed What to do if antibiotics cause nausea 0.23 0.35 Emily the scientist Coping strategies for nausea resulting from antibiotic use 0.00 0.51 Noah the curious child How can I feel better if medicine for sickness makes me throw up ? 0.19 0.10 Non-native What should one do when antibiotics make feel sick 0.07 0.13 Voice Please find solutions for dealing with nausea caused by antibiotic medication 0.26 0.63 Paraphrasing What to do when antibiotics make you queasy 0.35 0.50 Naturality Treatment for nausea from antibiotics 0.15 0.47 2. Related Work We examine two aspects of past work. 2.1. Users and Search Queries The relationship between user characteristics, context, and search behaviour has been widely studied. For example, mobile search queries are often shorter than desktop ones, likely due to limited input options on mobile devices ( Church et al. 2007 ) . The accuracy of these queries is also compromised by the fragmented attention of users, resulting in fewer hits and lower performance ( Harvey and Pointon 2017 ) . On the other hand, voice queries, which are increasingly prevalent, are generally longer and more closely resemble natural spoken language ( Guy 2016 ) . Language proficiency plays a crucial role in the success of online search queries. As highlighted in a qualitative study by Chu et al. 2015 , non-native English speakers faced challenges in formulating queries