Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks
Luke Guerdan †, Devansh Saxena †, Stevie Chancellor ∗, Zhiwei Steven Wu ∗, Kenneth Holstein ∗ · 2025
Read Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks by Luke Guerdan †, Devansh Saxena †, Stevie Chancellor ∗, Zhiwei Steven Wu ∗, Kenneth Holstein ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02819.txt.
Introduction
Luke Guerdan, Devansh Saxena, and co-authors study how data scientists turn fuzzy ideas into a target for a model. Their paper is from two thousand twenty-five. A target may be a proxy for a soft notion, such as true health need or real student work. How that proxy gets built is not well known.
They speak with data scientists in schools and health care. The picture that emerges is one of bricolage: people make do with the data at hand, in a craft-like way. They weigh truth, ease, forecast power, port across sites, and cost. They swap or merge outcomes when one fails. The authors point to ways tools and study can back this craft.
Catalog record
| CREATOR | Luke Guerdan †, Devansh Saxena †, Stevie Chancellor ∗, Zhiwei Steven Wu ∗, Kenneth Holstein ∗ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 107 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-02819
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks DOI: XXXXXXX.XXXXXXX Conference: Proc. ACM Hum.-Comput. Interact. 5, CSCW2; 2025; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 Thanks: † \dagger Co-first authors contributed equally to this research. ∗ Co-senior authors contributed equally to this research. CCS: Human-centered computing Empirical studies in HCI CCS: Computing methodologies Machine learning CCS: Computing methodologies Supervised learning Journal: PACMHCI Volume: 7 CSCW2 1 11 DOI: 10.1145/XXXXXXX Luke Guerdan †, Devansh Saxena †, Stevie Chancellor ∗, Zhiwei Steven Wu ∗, Kenneth Holstein ∗ Abstract . Data scientists often formulate predictive modeling tasks involving fuzzy, hard-to-define concepts, such as the “authenticity” of student writing or the “healthcare need” of a patient. Yet the process by which data scientists translate fuzzy concepts into a concrete, proxy target variable remains poorly understood. We interview fifteen data scientists in education (N=8) and healthcare (N=7) to understand how they construct target variables for predictive modeling tasks. Our findings suggest that data scientists construct target variables through a bricolage process, involving iterative negotiation between high-level measurement objectives and low-level practical constraints. Data scientists attempt to satisfy five major criteria for a target variable through bricolage: validity, simplicity, predictability, portability, and resource requirements. To achieve this, data scientists adaptively use problem (re)formulation strategies , such as swapping out one candidate target variable for another when the first fails to meet certain criteria (e.g., predictability), or composing multiple outcomes into a single target variable to capture a more holistic set of modeling objectives. Based on our findings, we present opportunities for future HCI, CSCW, and ML research to better support the art and science of target variable construction. 1. Introduction “Problem-solvers who do not proceed from top-down design but are arranging and rearranging a set of well-known materials can be said to be practicing bricolage. They tend to try one thing, step back, reconsider, and try another” (Turkle, p. 51). Data scientists grapple with subtle but important considerations when translating organizational goals into a formal modeling task. For example, consider a data science team developing a model to help physicians decide which patients have sufficient “healthcare need” for enrollment in a high-risk care management program (e.g., ( Obermeyer et al. 2019 ) ). While developing a model, data scientists may work with clinicians to understand how to translate risk predictions to actionable interventions — for example, by selecting an appropriate risk cut-off for enrollment recommendations. Data scientists may also carefully consider whether the patient populations represented in their training data adequately match the characteristics of patients encountered when the model is deployed. The process of problem formulation describes how data scientists navigate such translational concerns as they map a high-level goal into a tractable computational problem ( Passi and Barocas 2019 ) . Figure 1. An illustration of the relationship between the target variable and problem formulation. A prediction task describes a mapping from features to a target variable . A problem formulation connects a prediction task to high-level modeling goals, including factors such as how risk scores are converted to intervention recommendations and the target population at deployment time. One critical aspect of problem formulation is the process of target variable construction (Figure 1 ). Many outcomes of interest to data scientists — such as the “authenticity” of student writing or the “healthcare need” of patients — are latent, theoretical constructs that are not directly observable in data. To develop a model targeting these constructs, data scientists must construct a proxy variable based on outcomes that are readily available. For example, a data scientist might choose to predict patients’ cost of medical care or their number of recently diagnosed chronic health conditions as a proxy for their “healthcare need” ( Obermeyer et al. 2019 ) . Traditionally, measurement theory assumes that analysts follow a top-down workflow while mapping unobserved constructs to measurements in data ( Jacobs and Wallach 2021 ) . For example, a learning scientist might identify a latent construct they would like to measure among students (e.g., “math proficiency”) and design an operationalization (e.g., a test) before collecting data (e.g., student test scores). Disciplines in the quantitative social sciences have developed a rich set of guidelines (e.g., ( Van’t Veer and Giner-Sorolla 2016 ; Miller and Salkind 2002 ) ) and modeling tools (e.g., ( Everett 2013 ; Singh 2007 ) ) to help analysts implement this top-down measurement workflow. However, the traditional top-down conceptualization of measurement is in tension with our understanding of data science as a bottom-up process constrained by existing data ( Passi and Barocas 2019 ; Kross and Guo 2021 ) . Passi and Barocas 2019 observe that problem formulation in data science is an iterative process, where data scientists progressively reconcile their high-level objectives with low-level data and modeling constraints. As a result, while a traditional top-down measurement approach assumes that an analyst has the resources to collect data tailor-made to the goals of a study, data scientists are often forced to re-purpose the limited data that happens to be available. This disconnect creates a significant challenge: though many flawed data science projects can be traced back to target variable construction (e.g., ( Turque 2012 ; Amrein-Beardsley 2014 ; Mullainathan and Obermeyer 2021 ; Kawakami et al. 2022 ; Cheng et al. 2022 ) ), existing tools, frameworks, and pedagogical resources reflect a top-down planning approach towards measurement, rarely accounting for the bottom-up, resource-constrained reality of data science practice. To understand data scientists’ current practices and challenges in constructing target variables—and to identify opportunities to better support target variable construction in practice—in this work, we conducted semi-structured interviews with fifteen data scientists from education (N=8) and healthcare (N=7). We asked data scientists to walk us through how they navigate target variable construction in prior modeling projects. Participants also engaged in a target variable construction task, allowing us to observe their reasoning in a new modeling context. Our goal in these interviews was to understand how data scientists intertwine two conceptually distinct activities, measurement of latent constructs and prediction of target variables, during problem formulation ( Mussgnug 2022 ) . While prior empirical studies of data science practice have often centered prediction (e.g., ( Alspaugh et al. 2018 ; Wongsuphasawat et al. 2019 ; Kandel et al. 2012 ) ), we broaden our scope of inquiry to investigate the interplay between measurement validity and predictive utility in problem formulation. We find that data scientists construct target variables by leveraging creative and pragmatic approaches to make do with the limited data they have at hand . Data scientists craft their formulation through a process of iterative dialogue with their data, tweaking and refining their target variable until they conclude that it provides a sufficient operationalization for the unobserved construct of interest. To understand data scientists’ target variable construction practices, we draw upon the concept of bricolage , originally introduced by Claude Lévi-Strauss to contrast the rigid top-down methodology of Western science with the more tangible “science of the concrete” practiced in non-Western societies ( Levi-Strauss 1966 ) . Rather than implementing abstract concepts through top-down design, the bricoleur solves problems by arranging and re-arranging well-known materials in a bottom-up process. Like bricoleurs, data scientists in our study