Are Information Retrieval Approaches Good at Harmonising Longitudinal Survey Questions in Social Science? Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy. Proceedings of the 48th International
Wing Yan Li, Zeqiang Wang, Jon Johnson, Suparna De · 2025
Read Are Information Retrieval Approaches Good at Harmonising Longitudinal Survey Questions in Social Science? Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy. Proceedings of the 48th International by Wing Yan Li, Zeqiang Wang, Jon Johnson, Suparna De (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20679.txt.
Introduction
Search tools for matching survey questions across decades are weighed by Wing Yan Li and co-authors in a 2025 arXiv paper. Long studies reuse near-same questions, but word drift hides the pairs. The team frames a fresh search task: spot same-concept questions and their answer sets. A survey base from 1946 to 2020 supplies the test ground.
Plain odds models, light probes of language nets, and search-tuned nets all got a run. Search-tuned nets won overall, with other tools close behind. Re-ranking added at most a hair. Experts found the tools blind to word overlap with wrong sub-concepts.
Catalog record
| CREATOR | Wing Yan Li, Zeqiang Wang, Jon Johnson, Suparna De |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20679
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Are Information Retrieval Approaches Good at Harmonising Longitudinal Survey Questions in Social Science? Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy ISBN: 979-8-4007-1592-1/25/07 DOI: 10.1145/3726302.3730164 CCS: Computing methodologies Natural language processing CCS: Computing methodologies Unsupervised learning CCS: Computing methodologies Ensemble methods CCS: Computing methodologies Neural networks Wing Yan Li, Zeqiang Wang, Jon Johnson, Suparna De Abstract . Automated detection of semantically equivalent questions in longitudinal social science surveys is crucial for long-term studies informing empirical research in the social, economic, and health sciences. Retrieving equivalent questions faces dual challenges: inconsistent representation of theoretical constructs (i.e. concept/sub-concept) across studies as well as between question and response options, and the evolution of vocabulary and structure in longitudinal text. To address these challenges, our multi-disciplinary collaboration of computer scientists and survey specialists presents a new information retrieval (IR) task of identifying concept (e.g. Housing, Job, etc.) equivalence across question and response options to harmonise longitudinal population studies. This paper investigates multiple unsupervised approaches on a survey dataset spanning 1946-2020, including probabilistic models, linear probing of language models, and pre-trained neural networks specialised for IR. We show that IR-specialised neural models achieve the highest overall performance with other approaches performing comparably. Additionally, the re-ranking of the probabilistic model’s results with neural models only introduces modest improvements of 0.07 at most in F1-score. Qualitative post-hoc evaluation by survey specialists shows that models generally have a low sensitivity to questions with high lexical overlap, particularly in cases where sub-concepts are mismatched. Altogether, our analysis serves to further research on harmonising longitudinal studies in social science. 1. Introduction Large-scale longitudinal social surveys are fundamental to understanding societal changes across generations. Through questions tailored to specific populations and temporal contexts, surveys capture theoretical constructs, also referred to as concept or sub-concept, (e.g. housing conditions). Shifts in questionnaire structure, semantic frameworks, and concept usage due to shifting social contexts and target respondent characteristics (e.g. education level) mean that surveys vary in language, structure and response options. This poses a significant challenge when harmonising studies that span different time periods and populations. Table 1. Typology of survey questions with the corresponding concepts and/or sub-concepts highlighted. Bold texts are questions and italic texts are response options. Question Type Example Standard What is his current job ? 1. self-employed | 2. manager | 3. foreman 4. employee Qualified The thought of harming myself has occurred to me: 1. yes, quite often | 2. sometimes | 3. hardly ever | 4. never Compound What is she allergic to? (tick all that apply) dog 1. yes In this work, we address and investigate two main challenges when detecting conceptually equivalent questions in longitudinal surveys: (1) The linguistic challenge of capturing semantic equivalence despite surface-level variations in vocabulary and syntax. For instance, “dwelling” may evolve to “household” over time while maintaining the same underlying concept. (2) The structural challenge in which equivalent concepts may be distributed differently between question text and response options. In Table 1 , the sub-concept in the response categories of the standard question could fundamentally alter the concept being measured. Conversely, qualified questions do not capture any sub-concepts. While information retrieval (IR) techniques have advanced significantly in recent years, their applicability to survey harmonisation remains under explored. Traditional approaches like BM25 ( Robertson et al. 2009 ) excel at lexical matching but may miss semantic equivalences obscured by terminology shifts. Contemporary neural models ( Karpukhin et al. 2020 ; MacAvaney et al. 2019 ; Khattab and Zaharia 2020 ; Nair et al. 2022 ) while powerful at semantic modelling risk overfitting to superficial patterns without capturing the domain-specific nature of survey concepts. Moreover, existing work rarely addresses the unique requirements of structured survey questions where both question text and coded response options jointly define the measured construct. In this collaboration between computer scientists and survey specialists, we bridge this gap through three main contributions: (1) We formulate the task of detecting conceptual equivalence in longitudinal surveys as an IR problem, specifically addressing the challenge of integrating question and response text as coherent input sequences. (2) We present a systematic evaluation of unsupervised IR approaches on a dataset spanning 75 years (1946-2020) by comparing probabilistic models (BM25), linearly probed language models (DeBERTa, Qwen), and IR-specialised neural networks (M3-Embedding). (3) We combine quantitative metrics based on topic code alignment with qualitative analysis by domain experts to reveal both the promise and limitations of current IR methods for survey harmonisation. Our results reveal that while IR-optimised neural models achieve the highest overall performance ( F1=0.79 ), traditional approaches like BM25 remain competitive ( F1=0.75 ). Notably, qualitative analysis reveals that all models struggle with questions showing high lexical overlap but divergent sub-concepts, suggesting the need for more sophisticated approaches to concept granularity. 2. Related Work Computational Social Science Survey data harmonisation is the process of making different surveys more comparable and consistent across various variables such as studies, concepts, and time. Existing approaches rely heavily on classification schemas that are manually annotated by experts. These methods are typically statistical or procedural based ( Dubrow and Tomescu-Dubrow 2016 ) , thus leading to challenges (e.g. language barriers) when conducting cross-cultural research ( Tsai et al. 2024 ) . On the other hand, ( Stöhr 2024 ) proposed to train domain-specific Word2Vec embeddings using social science literature. Their evaluation of various tasks such as text classification with data collected from a sociology dictionary showed promising results. Information Retrieval Models Traditional IR continuous to adopt probabilistic-based methods that are based on statistical evidence (e.g. term-frequencies ( Robertson et al. 2009 ; Salton and Buckley 1988 ) ). Recently, neural IR models have garnered increasing attention by performing searches based on the underlying semantic information. Pre-trained language models like DeBERTa ( He et al. 2021 ) are commonly adopted as the reader model to embed sequences for a downstream retrieval model ( MacAvaney et al. 2019 ; Khattab and Zaharia 2020 ; Nair et al. 2022 ) . Some works also proposed IR-specialised models with various training strategies ( Guo et al. 2025 ; Li et al. 2023 ) and architectural designs ( Xiong et al. 2020 ; Chen et al. 2024 ) that aim at getting the best of both worlds (i.e. syntactic and semantic features). 3. Experiments Suppose there are n sets of questionnaires S={\{S_{1},S_{2},...,S_{n}\}} . Let U=\bigcup_{i=1}^{n}{S_{i}} be the union of all questionnaires. For each query question and the corresponding response options (q,r)\in{U} , the objective of the experiment is to identify all potential conceptually equivalent in U-\{(q,r)\} . Dataset We use questions from a large-scale longitudinal social science survey project consisting of 318 questionnaires. The questions possess diverse longitudinal spans ranging from 1946 to 2020. In total, there are 42,161 questions with three different typologies: standard, qualified , and compound questions, as illustrated in Table 1 . The standard questions have the most diverse distribution of concepts by presenting concepts and sub-concepts in the questions and response options, respectively. This introduces an inherent complexity and additional challenge since standard