Representativeness and response validity across nine opt-in online samples Authors: Michael N. Stagnaro, James N. Druckman, Adam J. Berinsky, Antonio A. Arechar, Robb Willer, David G. Rand Published: 2026 ABSTRACT The study of human behaviour has shifted in the past 15 years. It increasingly relies on opt-in non-probability online data sources. Here we offer an analysis of nine such sources (N = 13,053), aiming to inform researchers conducting experiments or correlational studies. We assess response validity (attentiveness, effort, honesty, speeding and attrition), the extent to which samples represent the underlying population (demographics, attitude representativeness and experimental treatment effects) and professionalism (number and frequency of studies taken and modality of device on which studies are taken). We document substantial variation across samples on each dimension. Samples using demographic quotas display relatively higher amounts of representativeness across multiple indicators (beyond demographics) but often exhibit less response validity. However, inclusion of two attention checks early in a survey can enhance response validity without negatively impacting representativeness. We offer guidance for choosing opt-in samples, depending on the purpose of the research and resource constraints. Main The past 15 years have seen a radical shift in how data about human attitudes and decision-making have been collected1. Of note has been the rise of opt-in non-probability data sources where respondents are recruited via various online sources such as advertisements, websites offering rewards or self-enrolment2–4. Understanding variation in responses across data sources (for example, platforms and samples) helps inform the interpretation of social and political signals derived from such data5,6. It also provides insight into a central data source that underlies research and policy decisions7. Considerable scholarship explores the extent to which opt-in samples are attentive (one aspect of response validity)8–12, or emulate the representative characteristics of gold-standard (more expensive) probability samples13–19. Here, we evaluate nine opt-in samples used when the goal is to make inferences about relationships between variables, often with experiments20. We include various representativeness metrics including measures of attitudes and experimental treatment effects, and we explore ways to maximize response validity without sacrificing representativeness. We find that the platforms widely vary in their performance. Thus, opt-in samples should not be grouped into a single category. Samples that use demographic quotas display relatively higher amounts of representativeness across multiple (even non-demographic) indicators but also often exhibit less response validity. The lower levels of response validity reflect, at least in part, increased mobile phone usage among the more representative samples. Moreover, we show that the inclusion of two attention checks early in a survey enhances response validity without substantially reducing representativeness. We conclude with a discussion of how to choose between opt-in samples, depending on the purpose of the research and resource constraints. We situate our work by first differentiating between what Baker et al.21 call ‘describers’ and ‘modelers’. The former focuses on estimating a population characteristic, while the latter attends to the relationships between variables (often, but not always, with experimental methods22). It is well established that non-probability samples, even with adjustments such as quotas or poststratification weights, do not fare well relative to probability samples for descriptive inference3,16,20,23,24 (although there is occasional disagreement with this assessment25). However, our central focus lies with the second type of data usage: relationships, often involving experiments. This is highly consequential given that opt-in data have ostensibly become the major source of data for this work26. For research on relationships such as experiments, there are at least two desired properties. The first is response validity: “the degree to which assessment data were obtained under conditions of appropriate motivation, adequate effort, and honesty on the part of clients/ respondents”27. This includes attentiveness and related constructs including the extent to which participants expend effort, honesty and respondents’ willingness to take time and persist through the entire study. While there are alternative ways to gauge response validity, such as evaluating open-ended responses, our general approach aligns with many survey studies that nearly always include attention checks18. Response validity notably matters for online data collections28. Indeed, factors such as attention and honesty are particularly crucial for inferences about relationships when experiments are involved. In such cases, the goal is to ensure that respondents take on particular values of the independent variable to assess its impact on the dependent variable29. In sharp contrast to descriptive inferences (for example, an election poll), one does not consider including inattentive respondents under the guise of such people invariably existing in the population. Kane30 explains, “attentiveness is often a precondition for having an efficacious [experimental] treatment” (emphasis in original)22,31,32. Inattentiveness can also skew factual manipulation checks in experiments30 and lead to response-set bias in observational studies33. The other desired property is sample representativeness: the extent to which the demographic characteristics of the participants match the population of interest (for example, the USA), the extent to which individuals hold the same attitudes as those in the target population, and the extent to which participants display similar patterns of experimental treatment effects compared with the target population. Although evaluating representativeness with opt-in samples carries the inevitable limitation that only observable measures can be assessed, it remains important, as most studies aim to generalize to a heterogeneous population. A more representative sample has at least two advantages for inferences about relationships (for example, causal inferences). First, while much work focuses on the sign generalizability of an effect (that is, a positive or negative effect), some work seeks insight into the magnitude (size) of effects, and, in this case, having a representative sample is crucial22,34,35. Second, identifying heterogeneous effects requires sufficient variation in key moderators. Some research suggests widespread homogeneity1,36,37; however, that work explores a limited number of moderators and may often lack power to document heterogeneous effects38,39. Indeed, Bryan et al.40 state that “most treatment effects are heterogeneous.” In addition to these two primary dimensions (response validity and representativeness), we evaluate professionalism: the frequency of taking surveys28, experience with certain types of survey, and the device on which the survey is taken. We include these items because they can influence response validity9 and/or representativeness41. For example, frequent survey-takers who sit at a computer taking surveys in succession may not reflect the population of interest; they may also have been previously exposed to manipulations that could contribute to task-specific ‘immunity’ (for example, familiarity with some paradigms42). Alternatively, taking a study on a mobile phone could lessen response validity given the screen size and likelihood of multitasking. How a given sample performs depends on an intersection of vendor and investigator decisions. The vendor manages recruitment, panel maintenance and payment structure; the investigator can use attention checks, apply quotas, and vary incentives. There are dozens of samples from which to choose. We selected samples most familiar to the author group (that includes researchers from cognitive science, economics, political science, psychology and sociology) and commonly used in their extended academic networks. These samples also collectively offered variance in quotas (as typically used), panel maintenance and platform oversight. We included nine different opt-in samples, collecting data from June 2022 to October 2023 (Supplementary Section 1). The platforms/samples are displayed in Table 1, where ‘NR’ for two of the samples refers to ‘nationally representative’—a term used by the platforms at the time of data collection to characterize the use of more quotas relative to their less representative versions (see the Methods for more details).Table 1Details of samplesSampleN% of dataDemographic quotasCR Toolkit1,0888.34NoneConnect1,0468.01NoneConnect NR1,0287.88Age, gender, race, ethnicityForthright1,1158.54Age, gender, race, ethnicity, geographic regionLucid4,50534.51Age, gender, race, ethnicity, geographic regionOpen M-Turk1,1038.45NoneProlific1,0518.05GenderProlific NR1,0407.97Age, gender, ethnicityStanford CRSTAL M-Turk panel1,0778.25NoneThe N for Lucid is substantially larger due to anticipated high levels of inattention. Table 1Details of samplesSampleN% of dataDemographic quotasCR Toolkit1,0888.34NoneConnect1,0468.01NoneConnect NR1,0287.88Age, gender, race, ethnicityForthright1,1158.54Age, gender, race, ethnicity, geographic regionLucid4,50534.51Age, gender, race, ethnicity, geographic regionOpen M-Turk1,1038.45NoneProlific1,0518.05GenderProlific NR1,0407.97Age, gender, ethnicityStanford CRSTAL M-Turk panel1,0778.25NoneThe N for Lucid is substantially larger due to anticipated high levels of inattention. Details of samples The N for Lucid is substantially larger due to anticipated high levels of inattention. All participants took the same survey. The survey included demographic items; attitude, belief and cognitive measures; several experiments; and items asking about participants’ platform and survey experiences (for details on the survey flow, see Supplementary Section 2). Throughout the survey, we included multiple attention checks and an effort question (described in detail below). Participants were not excluded for failing attention checks, as we will explore how outcomes change based on different approaches to filtering (removing those who miss attention checks). Question wording is available at 10.7910/DVN/E8YVSO. We set compensation to the sample provider-specific standard. We did this purposefully because altering payment outside of platform norms could lead to atypical respondents or responses. Response validity We measure response validity with (1) seven attention check items, distributed across the survey (with a total score of 0 = missed all to 1 = all correct); (2) one effort task, involving reading a paragraph of generic text followed by answering a question about it (0 = no/incorrect answer and 1 = correct answer)12; (3) one item at the end of the survey asking about honesty (0 = reported lying, 0.5 = reported not recalling, 1 = reported honesty); (4) a measure of speeding, identifying those who completed the study in less than 10 min (which roughly aligns with the fastest 10% of completion time28) (0 = speeding, 1 = not speeding); and (5) a measure of attrition, defined as entering the experimental section but not finishing the study (0 = attrited, 1 = finished). We then create an aggregate response validity measure by averaging all of the above items. We discuss scaling of the overall measure below. Representativeness As a benchmark against which to compare each sample’s demographic and attitudinal characteristics, we use data from the 2020 American National Election Studies online probability (weighted) sample (Supplementary Section 3). For demographic representativeness, we examine age, income, education, race/ethnicity, political party affiliation and religious identity. Each characteristic is coded to make the measure commensurate between the probability sample and our samples. We calculate the absolute difference in percentage points between the given platform’s sample and the benchmark at each possible variable response level (or category level). We then sum these deviations across variable levels (response categories)3. Specifically, for each scale point for each variable (v) on each platform sample (P), we take the absolute difference between the percentage of participants selecting that scale point (pS) and the percentage of the probability sample comparison (pPS) who chose the same option:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|{\mathrm{pS}}_{v1}-{\mathrm{pPS}}_{v1}|.$$\end{document}|pSv1−pPSv1|. We then take the overall average of the summed absolute difference for all variables:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bar{x}}_{{{\rm{p}}}_{1}-{{\rm{p}}}_{n}}\left(\sum ^{n}{{{v}}}_{1}-{{{v}}}_{n}\left(|{\mathrm{pS}}_{{{{v}}}_{1}}-{\mathrm{pPS}}_{{{{v}}}_{1}}|\right)\right).$$\end{document}x¯p1−pn∑nv1−vn∣pSv1−pPSv1∣. For an example, consider education; the 2020 American National Election Studies includes eight levels that we combine into six: ‘Less than high school credential’ (8.3%), ‘High school diploma/GED’ (27.7%), ‘Some college, no degree’ (18.5%), ‘Associate degree’ (10.4%), ‘Bachelor’s degree’ (22.6%) and ‘Graduate degree (Master’s, PhD and so on)’ (12.5%). For ‘Less than high school credential’, the Centralized Research Systems, Tools and Labor (CRSTAL) sample has 0.47% of participants at this level. We take the absolute difference with the probability sample benchmark to obtain a difference of 7.83 (|8.3–0.47|) percentage points. We sum this and the absolute difference (AbsDiff) for the other education levels (‘High school diploma/GED’: AbsDiff 18.35, ‘Some college, no degree’: AbsDiff 2.43, ‘Associate degree’: AbsDiff 2.5, ‘Bachelor’s degree’: AbsDiff 15.9, ‘Graduate degree (Master’s, PhD and so on)’: AbsDiff 5.35) to arrive at a summed absolute difference of 52.36 percentage points. We do the same for age, income, race/ethnicity, religious affiliation and political party, and take the overall average: an aggregate demographic representativeness score. Higher values indicate a larger deviation from the benchmark and, thus, a less representative sample. For attitude representativeness, we explore opinions about the death penalty, gun control, immigration, military spending, theist belief (that is, whether the respondent has a religious identity), political ideology, trust in people, reported voting behaviour in 2016 and trust in the federal government. We follow the same approach as just described (for example, summing absolute differences and taking the overall average) to generate an aggregate attitude representativeness measure, broken down by political party affiliation. We will also report deviations on individual items for both demographics and attitudes. In assessing representativeness, we can only study observable (for example measured) variables, and our use of the term ‘representative’ should be understood as such. The potential for unmeasured confounders is one reason why researchers turn to probability samples, particularly when the study’s purpose is to arrive at a population point estimate16. As mentioned, much of the work that uses the platforms we study here focuses on inferences about relationships, where probability samples may be less essential22. We thus also evaluate whether each sample could reproduce two experiments. The first asked respondents to assess spending for (randomly assigned) ‘helping the poor’ or ‘welfare’43–45. The second (risky framing effect experiment) had respondents choose between a concrete or probabilistic programme to address a disease, randomly varying whether they are framed in terms of gains (saving lives) or losses (people dying)46–49. Both experiments have been widely tested and replicated50. In addition, we explore hypothesized heterogeneous treatment effects to evaluate whether each sample contains sufficient variance on a given moderator34. We measure professionalism by asking how many similar studies on ‘attitudes and beliefs’ participants have completed through the given sample provider. We also asked participants what they were doing immediately before taking this survey (for example, completing a series of back-to-back studies versus working, socializing, shopping and so on). We also record which device the participant was using (mobile phone versus computer). The idea here is that ‘professionals’ are more likely to be taking many studies in succession on a computer (for example, as a job). Whether the presence of professional respondents is positive or negative depends on the purpose of the data collection51. Results Response validity Figure 1 shows average overall response validity as well as averages for each separate measure across samples. From these results, three groups of samples can be differentiated. The highest-validity group includes the two CloudResearch Connect samples, the two Prolific samples and the CRSTAL panel. These are notably high and functionally indistinguishable from each other (P > 0.05 for all comparisons within this set; P < 0.001 for all other comparisons with the lower-performing samples) (all tests here and across all results in this Article use two-tailed tests). Forthright and, to a lesser extent, CloudResearch’s M-Turk Toolkit (CR Toolkit) occupy the middle validity range; although they significantly differ from each other (P < 0.001), they can be clearly separated from the first group. Finally, validity is much lower on Lucid and Open Mechanical Turk (Open M-Turk). Lucid scores the lowest on all subcomponents other than honesty (which is lower on Open M-Turk); with both Lucid and Open M-Turk scoring significantly lower than Forthright and CR Toolkit (P < 0.001). The findings echo other work that reports higher validity for Prolific and CloudResearch relative to Open M-Turk9,19.Fig. 1Variation in response validity across samples.Violin plots depicting the distribution of overall validity by sample (higher scores indicate greater response validity). The dotted red line indicates the overall average across all samples. Red diamonds represent mean overall response validity; blue diamonds represent mean overall response validity after filtering on the first two attention checks. Other markers indicate the average for each subcomponent of response validity: aggregate of seven attention checks, effort, honesty, speeding and attriting (all calculated with no front-end filtering). The white dot in the violin plot indicates the median, and thicker lines represent the interquartile range (n = 13,053). Variation in response validity across samples. Violin plots depicting the distribution of overall validity by sample (higher scores indicate greater response validity). The dotted red line indicates the overall average across all samples. Red diamonds represent mean overall response validity; blue diamonds represent mean overall response validity after filtering on the first two attention checks. Other markers indicate the average for each subcomponent of response validity: aggregate of seven attention checks, effort, honesty, speeding and attriting (all calculated with no front-end filtering). The white dot in the violin plot indicates the median, and thicker lines represent the interquartile range (n = 13,053). How much can simple attention filters at the outset of the study (to screen out bots and highly inattentive participants before treatment) help with response validity? We find that excluding participants who fail either of two (extremely easy: ‘Please write “twenty-five” using numbers.’ and ‘Help us keep track of who is paying attention. Please select “Somewhat disagree” from the options below.’) initial attention checks substantially increases overall response validity (across samples, prefiltering mean 0.771, standard deviation (s.d.) 0.255, 95% confidence interval (CI) 0.77–0.78; postfiltering mean 0.822, s.d. 0.207, 95% CI 0.8–0.83), P < 0.001. It does not, however, meaningfully change the rank ordering across all samples (Fig. 1, blue diamonds). The exception is that Prolific and CRSTAL swap positions (differences for the top-performing samples are further highlighted in Supplementary Section 4). For the overall response validity score here, we remove the first two attention checks. The improvement in response validity raises the question of whether performance on the two front-end attention checks identifies (and subsequently removes) a lower-validity ‘type’ of participant52. To shed light on this question, we looked at Cronbach’s alpha scores for the full set of seven attention check items. Interestingly, there is comparatively little relationship in performance across attention checks for most of the six highest-validity samples (Connect: α = 0.26, Connect NR: α = 0.46, CR Toolkit: α = 0.19, CRSTAL: α = 0.28, Prolific: α = 0.05, Prolific NR: α = 0.05); however, we find much more consistency across attention checks in the three samples with lowest response validity (Forthright: α = 0.64, Lucid: α = 0.73, Open M-Turk: α = 0.58). This suggests that weeding out inattentive participants in the latter three samples could remove an inattentive type of participant and, thus, yield substantial increases in validity. Notably, the alpha scores for even these three latter samples are not particularly high, suggesting that treating our checks as a scale of attention is unreliable. This, however, is not a concern for us, as we use the scale to capture/summarize a range of attention/response validity indicators. We are identifying a type rather than offering a broad instrument to measure general attention. One might be concerned that these alpha values reflect another factor, such as participants’ familiarity with attention checks in surveys. We thus explore whether passing the front-end checks predicts overall response validity (calculated without the first two attention checks). If performance on attention checks primarily reflects familiarity with how such screeners work, it should have little association with overall response validity, which encompasses many items, including actual behaviours (for example, speeding, attrition, effort expended and so on). Yet, we find clear associations: whether participants passed both front-end checks has meaningful predictive value for overall response validity on the three samples with lower response validity (Forthright: β = 0.48, R2 = 0.23, P < 0.001, b = 0.29, 95% CI 0.26 to 0.33; Lucid: β = 0.61, R2 = 0.38, P < 0.001, b = 0.37, 95% CI 0.35 to 0.38; Open M-Turk: β = 0.49, R2 = 0.24, P < 0.001, b = 0.29, 95% CI 0.26 to 0.32). It has less of a relationship for the six samples with higher response validity as they are already at a ceiling: Connect: β = 0.29, R2 = 0.08, P < 0.001, b = 0.21, 95% CI 0.17 to 0.25; Connect NR: β = 0.38, R2 = 0.14, P < 0.001, b = 0.2, 95% CI 0.17 to 0.23; CR Toolkit: β = 0.40, R2 = 0.16, P < 0.001, b = 0.35, 95% CI 0.3 to 0.39; CRSTAL: β = 0.37, R2 = 0.14, P < 0.001, b = 0.27, 95% CI 0.23 to 0.31; Prolific: β = 0.23, R2 = 0.05, P < 0.001, b = 0.17, 95% CI 0.13 to 0.21; Prolific NR: β = 0.30, R2 = 0.09, P < 0.001, b = 0.17, 95% CI 0.14 to 0.21. This evidence strongly suggests that early attention checks substantially help, particularly with samples that are otherwise low in validity. The high alphas are not simply capturing familiarity but rather a low response validity ‘type’. We next quantify the validity gains from implementing different numbers of attention checks, because studies vary in how many checks they tend to include30,52. Table 2 presents the average overall response validity score for each sample when limiting it to those who answered each number of attention check items correctly (from passing none or ‘no filter’ to passing all seven or ‘all filters’). These attention checks were all short and easy to answer, and had no trick/deceptive aspects (see Supplementary Section 5 for wording)53. We see that it is possible to achieve high response validity on all samples with a sufficiently high level of filtering, except on Open M-Turk (the awkward decreases at 2 + 4 are a statistical artefact, as explained in Supplementary Section 6), although this leads to a substantial loss of participants for many samples (especially Lucid). In Supplementary Section 7, we show honesty and effort results along these same lines. Higher rates of filtering may also undermine the observed representativeness of a given sample, a question to which we next turn.Table 2Means of overall response validity and percentage of participants retained for each sample as a function of increasing attention filteringAverage response validityMinimum number of attention checks passed (response validity and percentage of sample)No filterFront 2Front 2 + 1Front 2 + 2Front 2 + 3Front 2 + 4All filtersLucid0.65100%0.7270%0.7570%0.7466%0.8152%0.7839%0.9414%Open M-Turk0.77100%0.7687%0.7887%0.7884%0.7979%0.7464%0.8426%Forthright0.84100%0.8889%0.9089%0.9087%0.9381%0.8570%0.9729%CR Toolkit0.87100%0.8995%0.9295%0.9392%0.9589%0.8681%0.9635%Connect0.90100%0.9397%0.9597%0.9696%0.9694%0.8686%0.9835%CRSTAL0.90100%0.9497%0.9697%0.9696%0.9793%0.8787%0.9936%Prolific0.90100%0.9397%0.9597%0.9695%0.9793%0.8785%0.9938%Prolific NR0.91100%0.9496%0.9696%0.9795%0.9694%0.8786%0.9836%Connect NR0.91100%0.9595%0.9795%0.9795%0.9794%0.8787%0.9936%Total0.85100%0.8891%0.9091%0.9190%0.9285%0.8476%0.9632%Scores are calculated using the following procedure: we first report raw validity scores, then we report validity scores filtering on those who answered the front two simple attention checks correctly. Next, we report validity scores for those who correctly answered both of the simple checks, plus any additional check correctly; then the front checks plus any two additional checks; and so on. Importantly, this comparison involves using a custom validity score at each level, such that it removes the attention check item being set to ‘correct’ and averages across all variations when evaluating the overall score to prevent autocorrelations. The second column of each grouping is the percentage of the original sample used, indicating the amount lost at each level of filtering. Table 2Means of overall response validity and percentage of participants retained for each sample as a function of increasing attention filteringAverage response validityMinimum number of attention checks passed (response validity and percentage of sample)No filterFront 2Front 2 + 1Front 2 + 2Front 2 + 3Front 2 + 4All filtersLucid0.65100%0.7270%0.7570%0.7466%0.8152%0.7839%0.9414%Open M-Turk0.77100%0.7687%0.7887%0.7884%0.7979%0.7464%0.8426%Forthright0.84100%0.8889%0.9089%0.9087%0.9381%0.8570%0.9729%CR Toolkit0.87100%0.8995%0.9295%0.9392%0.9589%0.8681%0.9635%Connect0.90100%0.9397%0.9597%0.9696%0.9694%0.8686%0.9835%CRSTAL0.90100%0.9497%0.9697%0.9696%0.9793%0.8787%0.9936%Prolific0.90100%0.9397%0.9597%0.9695%0.9793%0.8785%0.9938%Prolific NR0.91100%0.9496%0.9696%0.9795%0.9694%0.8786%0.9836%Connect NR0.91100%0.9595%0.9795%0.9795%0.9794%0.8787%0.9936%Total0.85100%0.8891%0.9091%0.9190%0.9285%0.8476%0.9632%Scores are calculated using the following procedure: we first report raw validity scores, then we report validity scores filtering on those who answered the front two simple attention checks correctly. Next, we report validity scores for those who correctly answered both of the simple checks, plus any additional check correctly; then the front checks plus any two additional checks; and so on. Importantly, this comparison involves using a custom validity score at each level, such that it removes the attention check item being set to ‘correct’ and averages across all variations when evaluating the overall score to prevent autocorrelations. The second column of each grouping is the percentage of the original sample used, indicating the amount lost at each level of filtering. Means of overall response validity and percentage of participants retained for each sample as a function of increasing attention filtering Scores are calculated using the following procedure: we first report raw validity scores, then we report validity scores filtering on those who answered the front two simple attention checks correctly. Next, we report validity scores for those who correctly answered both of the simple checks, plus any additional check correctly; then the front checks plus any two additional checks; and so on. Importantly, this comparison involves using a custom validity score at each level, such that it removes the attention check item being set to ‘correct’ and averages across all variations when evaluating the overall score to prevent autocorrelations. The second column of each grouping is the percentage of the original sample used, indicating the amount lost at each level of filtering. Representativeness We evaluate the representativeness of our samples by comparing them against a probability sample benchmark (Supplementary Sections 3 and 8) for (1) demographic composition and (2) attitude similarity. Although the obvious intuition is that samples with (more) quotas will be more representative, this is not automatically the case because the metrics are not one-to-one matches with the quotas (and, of course, we do not use any kind of attitudinal quota). For demographic and attitude representativeness, we use the metric discussed in the ‘Representativeness’ section: the summed (response category) absolute deviations measure, averaged across items. Lower scores indicate being more representative.This metric should not be interpreted as a pure ‘percentage deviation’, as it lacks a natural scale—it sums deviations across categories, and these sums can exceed 100% (Supplementary Section 8). For attitude representativeness, we differentiate respondents by partisanship (counting leaning partisans as partisans) given that it is well documented that partisanship shapes these attitudes. Figures 2 and 3 show the results, respectively. In Supplementary Section 9, we report the direction (for example, liberal or conservative) of the attitude deviations; these are not evident in Fig. 3 because the metric uses the absolute deviation.Fig. 2Deviation in demographics from probability sample benchmarks for each sample.Bars indicate the average deviation from the probability sample benchmarks for each demographic across samples. Symbols depict political party affiliation, religious identification, income, race/ethnicity, age and education level. The dotted red line shows the average deviation collapsing across all samples. The y axis shows averaged absolute deviations in percentage points. These values do not fall on a defined finite scale (for example, 0 to 100); see the text for a discussion. As this is sample-level data, no error bars are presented, nor are any inferential statistics performed.Fig. 3Deviation in attitude representativeness from a probability sample benchmark for each sample among Democrats, Independents and Republicans.Bars indicate the overall comparison, while each symbol disaggregates this score and shows the deviation for political ideology, military spending, gun control, death penalty, immigration, trust in the federal government, voting in 2016, trust in people and theist belief. Dotted red lines indicate average absolute deviation collapsing across all samples. From left to right, plots represent scores for Republican plus Republican leaners, pure Independents, and Democrats plus Democrat leaners. The y axis shows average absolute deviations in percentage points. These values do not fall on a defined finite scale; see the text for a discussion. As this is sample-level data, no error bars are presented, nor are any inferential statistics performed. Deviation in demographics from probability sample benchmarks for each sample. Bars indicate the average deviation from the probability sample benchmarks for each demographic across samples. Symbols depict political party affiliation, religious identification, income, race/ethnicity, age and education level. The dotted red line shows the average deviation collapsing across all samples. The y axis shows averaged absolute deviations in percentage points. These values do not fall on a defined finite scale (for example, 0 to 100); see the text for a discussion. As this is sample-level data, no error bars are presented, nor are any inferential statistics performed. Deviation in attitude representativeness from a probability sample benchmark for each sample among Democrats, Independents and Republicans. Bars indicate the overall comparison, while each symbol disaggregates this score and shows the deviation for political ideology, military spending, gun control, death penalty, immigration, trust in the federal government, voting in 2016, trust in people and theist belief. Dotted red lines indicate average absolute deviation collapsing across all samples. From left to right, plots represent scores for Republican plus Republican leaners, pure Independents, and Democrats plus Democrat leaners. The y axis shows average absolute deviations in percentage points. These values do not fall on a defined finite scale; see the text for a discussion. As this is sample-level data, no error bars are presented, nor are any inferential statistics performed. The demographic figure shows the two samples using the most quotas (Forthright and Lucid) are more representative, followed by those with fewer quotas (Connect NR and Prolific NR)19, and then those with no quotas (with Open M-Turk being notably poor). As these analyses yield only a single observation per sample, we do not conduct inferential statistics but simply provide the rank ordering. The attitude results follow a similar pattern but with more nuance. Although the attitude measures in the four samples with multiple quotas clustered more than the demographic measures, they still fall below the overall mean line, suggesting some interesting variation. Specifically, Prolific NR does not perform better than the non-quota samples for Independents; Forthright does notably well for Democrats and Independents; and the four samples using any quotas are almost indistinguishable from the others for Republicans. Overall, these samples seem to do better with Democrats and worse with Republicans, in line with concerns raised by work on polling54. It is also striking that there are some topics where almost all samples notably deviate from the probability sample benchmark (for example, gun control). Why this occurs requires further investigation. Considering these results along with those presented in the prior section, we see that, at least for our samples, the ones that score lower on response validity are more representative (for example, Lucid and Forthright). (The exception is Open M-Turk, which scored poorly on all metrics.) This is sensible insofar as samples that seek out a wider range of respondents may include more respondents who are less attentive. For now, we turn to the consequences of filtering out inattentive participants for representativeness30,52. Figure 4 shows that, perhaps surprisingly, there is little change in average demographic or attitude representativeness across all but the most extreme levels of attention filtering (see Supplementary Section 10 for sample sizes underlying Fig. 4). This suggests that deviations reflect the platforms having atypical respondents rather than stemming from inattentiveness specifically. That said, averaging these results hides important variation, most notably in the Lucid sample regarding demographics (Supplementary Fig. 11-3). Here, we see that filtering beyond two attention checks introduces substantial and sometimes unexpected deviations. For example, the sample becomes more deviant in terms of education and age but less so for income, highlighting that it no longer resembles the original draw. In Supplementary Section 11, we present results for the individual demographics across all other samples as well as for the other response validity constructs; in Supplementary Section 12, we do the same for all samples (including Lucid) for the individual attitudinal items. Deviations increase as more extreme filtering is applied, but in no case is there meaningful (significant) change with two attention checks (and, for many samples, this holds for up to five filters). Lucid is generally the most sensitive. The key insight is that these data suggest that including a limited number (for example, two to four with samples other than Lucid) of early attention filters substantially increases response validity (Table 2) with no loss in representativeness (Fig. 4).Fig. 4Response validity can be improved by filtering on attentiveness without (mostly) compromising representativeness.The three plots on the right depict average deviation in attitude representativeness for each sample, across levels of attention filtering. Attention levels reflect filtering based on the minimum number of attention checks passed, with ‘0’ representing the full sample and ‘7’ representing participants who passed all seven attention checks. These averages represent composers of several subcategories that themselves are fairly stable across filtering, with the notable exception of Lucid (Supplementary Sections 8 and 9). As this is sample-level data, no error bars are presented, nor are any inferential statistics performed. Response validity can be improved by filtering on attentiveness without (mostly) compromising representativeness. The three plots on the right depict average deviation in attitude representativeness for each sample, across levels of attention filtering. Attention levels reflect filtering based on the minimum number of attention checks passed, with ‘0’ representing the full sample and ‘7’ representing participants who passed all seven attention checks. These averages represent composers of several subcategories that themselves are fairly stable across filtering, with the notable exception of Lucid (Supplementary Sections 8 and 9). As this is sample-level data, no error bars are presented, nor are any inferential statistics performed. Experimental effects We next consider experimental treatment effects across samples. Some scholars suggest that failed replications come from using non-representative samples55,56, with insufficient heterogeneity57. This result could occur due to variation in treatment effect size across different subgroups (for example, smaller effect sizes in younger participants that reduce the average treatment effect when they compose a sizeable part of the sample), or atypical attitudes of particular groups (for example, a sample of mostly fiscally conservative but socially liberal Republicans leading to a different average treatment effect than a sample of more typical Republicans). We therefore included two canonical experiments: (1) contrasting the effect of framing a policy as ‘helping the poor’ versus ‘increasing welfare’, and (2) participants’ preference for concrete gains over probabilistic losses (that is, a risky choice framing effect). All results below do not filter on any attention checks; however, we present results with attention filtering in Supplementary Section 13. Also, as an aside, we do not find evidence of differential attrition or differential response validity in either of the experiments on any sample. ‘Helping the poor’ versus ‘Those on welfare’ In this study, respondents are asked whether we are “spending too much, too little, or about the right amount…” when it comes to helping the poor (in one experimental condition) or those on welfare (in another experimental condition). Previous work, with probability samples, finds greater relative support for government spending when a policy is described as ‘helping the poor’43,45. This effect manifests to a much greater extent for Republicans compared with Democrats. Treating the response option as ordered (for example, from spending too little to spending too much), we use an ordinal logistic regression, We replicate the main effect collapsing across samples (b = −0.88, P < 0.001, 95% CI −0.96 to −0.81), and in each sample (Fig. 5a). However, the magnitude and precision varies across samples: the largest effect is with Forthright (b = −1.17, P < 0.001, 95% CI −1.4 to −0.91), the smallest effect is with Lucid (b = −0.77, P < 0.001, 95% CI −0.89 to −0.64), and the only sample showing no significant effect is Open M-Turk (b = −0.20, P = 0.09, 95% CI −0.43 to 0.03), possibly reflecting low levels of attention on that sample30.Fig. 5The effect of welfare framing on policy support.a, The overall main effect of policy frame on support broken down by sample (n = 11,584). b, The interaction coefficient between treatment and party affiliation (Republican/Democrat) (n = 9,776). c, The interaction coefficient between treatment and dichotomous race (white/non-white) variable (n = 11,584). d, The interaction coefficient between treatment and dichotomous age variable (above/below 40) (n = 11,572). e, The interactions coefficient between treatment and gender (female/male) (n = 11,488). Blue indicates the two high-representative samples consistently patterning together (Forthright and Lucid), purple indicates the mid-representative samples performing midway between both groups (Prolific NR and Connect NR) for attitude representativeness and demographic representativeness, and red depicts non-representative samples. Central dots in each coefficient represent raw coefficients, with the lines depicting 95% CIs all coming from an ordinal logistic regression using a two-tailed test. For ease of comparison, x axes are not consistent across panels. The effect of welfare framing on policy support. a, The overall main effect of policy frame on support broken down by sample (n = 11,584). b, The interaction coefficient between treatment and party affiliation (Republican/Democrat) (n = 9,776). c, The interaction coefficient between treatment and dichotomous race (white/non-white) variable (n = 11,584). d, The interaction coefficient between treatment and dichotomous age variable (above/below 40) (n = 11,572). e, The interactions coefficient between treatment and gender (female/male) (n = 11,488). Blue indicates the two high-representative samples consistently patterning together (Forthright and Lucid), purple indicates the mid-representative samples performing midway between both groups (Prolific NR and Connect NR) for attitude representativeness and demographic representativeness, and red depicts non-representative samples. Central dots in each coefficient represent raw coefficients, with the lines depicting 95% CIs all coming from an ordinal logistic regression using a two-tailed test. For ease of comparison, x axes are not consistent across panels. Adding partisanship and the interaction between treatment and partisanship to the model replicates the expected interaction (b = −0.45, P < 0.001, 95% CI −0.61 to −0.28)45; Republicans show a much stronger framing effect (b = −1.18, P < 0.001, 95% CI −1.3 to −1.0) compared with Democrats (b = −0.82, P < 0.001, 95% CI −0.93 to −0.71). Crucially, across samples, the interaction with partisanship is significant only in Lucid: (b = −0.75, P < 0.001, 95% CI −1.03 to −0.46 (Fig. 5b). We also find, consistent with Green and Kern45, that the more representative samples reveal moderation by age (above versus below 40; Fig. 5d). In addition, we find evidence of moderation by race (white versus non-white; Fig. 5c), concentrated in the more representative samples. We find no evidence of gender moderation in any sample (Fig. 5e). For a full list of results, see Supplementary Section 14. Overall, these results accentuate the importance of having a more representative sample to estimate heterogeneous effects, and presumably more accurate average treatment effects. It is worth pointing out as well that none of the samples uses quotas on partisanship, and thus, such a quota is not needed to identify moderation by partisanship. The results counter Druckman and Kam’s34 claim that heterogeneous effects can be discovered even with minimal variance on the moderator (for example, some of the samples with age quotas do not find the age moderation). Finally, we assess the robustness of the heterogeneous effects when adding pretreatment attention filters (Supplementary Section 13). The politics and age replication sustain with up to four attention checks. This substantiates the prior findings that adding limited filters does not undermine representativeness results. Risky choice framing effect experiment The risky choice framing effect experiment asks participants to choose between two programmes: programme A, where 200 people will be saved and 400 people will die (certain outcome), and programme B, where there is a 1/3 probability that 600 people will be saved and 0 people die and there is a 2/3 probability that 0 people will be saved and 600 people will die (probabilistic outcome). The frame is either gains (lives saved) or losses (dying) (see Supplementary Section 15 for details). Using a logistic regression, we predict the probability of selecting the probabilistic option over the concrete option as a function of frame (loss versus gain). We successfully replicate previous work, finding a significant effect of frame overall (b = 0.99, P < 0.001, 95% CI 0.88 to 1.10), and in each sample, with the largest effect being Prolific (b = 1.45, P < 0.001, 95% CI 1.07 to 1.8) and the smallest effects again being Open M-Turk (b = 0.627, P = 0.001, 95% CI 0.25 to 1.00) and then Lucid (b = 0.744, P < 0.001, 95% CI 0.55 to 0.94) (that is, samples with lower response validity) (Fig. 6a).Fig. 6Loss frame on policy selection.a, The overall main effect of policy frame on support broken down by sample (n = 5,798). b, The interaction coefficient between treatment and party affiliation (Republican/Democrat) (n = 4,885). c, The interaction coefficient between treatment and dichotomous race (white/non-white) variable (n = 5,798). d, The interaction coefficient between treatment and dichotomous age variable (above/below 40) (n = 5,790). e, The interactions coefficient between treatment and gender (female/male) (n = 5,748). Blue indicates the two high-representative samples consistently patterning together (Forthright and Lucid), purple indicates the mid-representative samples performing midway between both groups (Prolific NR and Connect NR) for attitude representativeness and demographic representativeness, and red depicts non-representative samples. Central dots in each coefficient represent raw coefficients, with the lines depicting 95% CIs all coming from a logistic regression using a two-tailed test. For ease of comparison, x axes are not consistent across panels. Loss frame on policy selection. a, The overall main effect of policy frame on support broken down by sample (n = 5,798). b, The interaction coefficient between treatment and party affiliation (Republican/Democrat) (n = 4,885). c, The interaction coefficient between treatment and dichotomous race (white/non-white) variable (n = 5,798). d, The interaction coefficient between treatment and dichotomous age variable (above/below 40) (n = 5,790). e, The interactions coefficient between treatment and gender (female/male) (n = 5,748). Blue indicates the two high-representative samples consistently patterning together (Forthright and Lucid), purple indicates the mid-representative samples performing midway between both groups (Prolific NR and Connect NR) for attitude representativeness and demographic representativeness, and red depicts non-representative samples. Central dots in each coefficient represent raw coefficients, with the lines depicting 95% CIs all coming from a logistic regression using a two-tailed test. For ease of comparison, x axes are not consistent across panels. Previous work has identified an interaction between frame and risk preferences49 (see more details in Supplementary Section 15), and, in line with this, we find a significant interaction (b = −0.145, P = 0.04, 95% CI −0.27 to −0.006), such that those below the median level of risk show a larger framing effect (b = 1.12, P < 0.001, 95% CI 0.96 to 1.28) compared with those above the median level of risk (b = 0.91, P < 0.001, 95% CI 0.75 to 1.05). The interaction effect is not statistically significant in any of the individual samples (Supplementary Section 15). Moreover, Fig. 6b–e reveals little evidence of any moderation between treatment and any of the demographic variables, and no evidence of significant variation across samples (heterogeneity chi-squared between P = 0.37 and P = 0.52). Also, unlike the other experiment, there is no obvious pattern whereby the more representative samples agree on an effect size or direction. (Given the lack of heterogeneous effects, we do not assess the impact of attention check filtering.) For a full list of results, see Supplementary Section 14. Treatment heterogeneity in the first experiment (helping the poor versus welfare) but not the second (risky choice framing effect) suggests that certain types of experiment may depend less on representative samples. This difference may hinge on the nature of the experimental content58. The first experiment involves social/political content, whereas the second has a cognitive focus. To further probe the role of sample representativeness when it comes to social/political content, we test for relationships that have been documented with probability samples. This includes the notably diminished relationship between political ideology and party affiliation for Black Americans as compared with the strong correlation for white Americans59 (Supplementary Section 25a), as well as the notable male bias in support for Donald Trump60 (Supplementary Section 25b). We find evidence of clear relationships only in the more representative samples. The apparent absence of relationships (or inverse relationships) in the less representative samples further highlights the risk of obtaining a non-representative sample when asking such questions, while also raising the possibility of respondents faking their identity affiliations (that is, bogus respondents)61. Platforms and users need to be vigilant in checking for such respondents as well as bots and artificial intelligence responses62. We also run an analysis in Supplementary Section 16 using 6 key demographics (race, age, education, gender, income and political affiliation) to predict the 22 main attitudes, beliefs and cognitive measures we collected. This provides the reader with exhaustive and unbiased comparisons (on observed variables). Across all relationships, we see some discrepancies between the more versus less representative samples (Supplementary Section 16). Sample weights One solution to the lack of sample representativeness is to apply poststratification weights. Weights mechanically improve representativeness on the variables weighted, but do not ensure more than that. MacInnis et al.23 offer an extensive comparison of non-probability and probability samples, concluding that “post stratification weights …only sometimes improved the accuracy of the nonprobability samples. Furthermore, weighting did not eliminate the superiority of the probability sample surveys over the nonprobability sample surveys…”16. For us, the question is whether using weights helps the less representative samples identify the heterogeneous effects in the helping the poor versus welfare experiment22,63. In Supplementary Section 17, we add weights to each sample; we find that the weights improve consistency for the main effect (even Open M-Turk replicates the main effect). Critically, however, there continue to be inconsistent moderated effects after weighting, such that the weights do not successfully recover treatment effect heterogeneity. This is the case even when we combine all the less representative samples in an attempt to counteract the loss of power that comes from weighting. Therefore, we find no evidence that weighting closes the gap between the more and less representative samples in terms of heterogeneous effects. In sum, if homogeneous effects are assumed a priori, the choice of sample matters little in terms of representativeness15. If there is the possibility of heterogeneous effects, weights are an imperfect solution, even putting power considerations aside. Our experimental results suggest that heterogeneous effects may occur more often with social/political content relative to a cognitive focus. Although we focused on demographic moderators, other variables may also be relevant in such settings, including factors specific to the topic under study, such as issue importance or motivation-, skill- and experience-based variables22,64. Professionalism We next examine variation across samples in levels of professionalism, by which we mean “well-trained or experienced survey-takers who seek out larger numbers of surveys…”51. We asked participants how many studies like ours they had participated in. We find wide variation in past experience, with Forthright (median 5) and Lucid (median 4) on the low extreme, and CRSTAL (median 300) and CR Toolkit (median 200) on the high extreme (Supplementary Section 26). Regressing response validity against the number of reported studies completed (log10-transformed), including sample dummies, shows a significant positive relationship (β = 0.12, P < 0.001). This result highlights the complexity of professional survey-takers: they differ from most in the population (for example, representative samples show less experience), but they offer more valid responses65. In the end, normative assessments of professionalism depend on the purpose of the survey51; that is, ‘professionalism’ is not inherently a positive or a negative. We additionally asked participants to report which activities they were engaged with before taking the survey. The activities include ‘completing other studies’, ‘working on the computer’ and a variety of other non-study-related activities (for example, shopping, socializing and so on). Participants could select as many options as appropriate. The idea is that completing other surveys or working on a computer is probably correlated with being a professional participant, whereas the other activities would suggest a lower likelihood of being a professional participant. In line with this reasoning, participants from the more representative samples were less ‘professional’ in their associated activity: less likely to be working on the computer or completing other studies before the current task (Supplementary Section 24). This coheres with the findings regarding the number of surveys (and shows a significant positive association with log10 previously completed surveys: β = 0.18, P < 0.001, b = 0.38, 95% CI 0.34 to 0.42). However, the results here are not as stark; for example, a similar percentage of Lucid respondents reported having just completed another survey as did Open M-Turk respondents, with Forthright showing the most diverse spread of activities (Supplementary Section 24). Mobile phones We also chart whether respondents completed the survey on a mobile phone. We treat this as a metric of professionalism because professionals would be better off working on a main workstation (for example, computer) rather than on their phone. We find a negative relationship between completing the survey on a mobile phone and the number of previously completed similar surveys (the correlation with the log10 of previously completed surveys is β = −0.10, P < 0.001, b = −0.23, 95% CI −0.28 to −0.18). This indicates that the canonical professional who takes many surveys is less likely to do so on a mobile phone. In terms of platforms, we find that Lucid and Forthright have a notably larger percentage of participants taking the study on a mobile phone (70.61% and 69.06%, respectively) compared with the other samples: Connect (9.66%), Connect NR (9.63%) CR Toolkit (4.78%), CRSTAL (7.61%), Open M-Turk (.54%), Prolific (20.46%) and Prolific NR (18.94%) (Supplementary Section 18). This is sensible insofar as the former two platforms presumably achieve greater representativeness by broadening their reach via mobile phones. Recall that these more representative samples have lower response validity. It seems that, for response validity, mobile phone usage may be more important than internal maintenance and curation of the panel (Forthright is internally maintained, and Lucid is not), although Open M-Turk would suggest otherwise. Even so, whether this result regarding response validity stems definitively from mobile phone usage is an important question; existing work offers mixed evidence on the validity of responses on mobile phones, relative to personal computers66–68. We add to this work by regressing a user’s response validity index on a dummy variable indicating whether the survey was taken on a mobile device as well as dummy variables for the samples. We find that, overall, response validity is significantly lower on mobile phones (β = −0.14, P < 0.001, b = −0.06, 95% CI −0.08 to −0.06)67. We additionally find evidence of heterogeneity. When collapsing the samples into the three levels of representativeness (high, middle and low), we see that the most representative samples (Forthright and Lucid) show a clear negative relationship between mobile use and response validity (β = −0.19, P < 0.001) while the middle-level samples (Prolific NR and Connect NR) reveal no relationship (β = −0.03, P = 0.23). Interestingly, the low (non-representative) samples have a slight positive relationship (β = 0.08, P < 0.001) (Supplementary Section 18). When we use two front-end attention filters, response validity improves in these samples. Specifically, we regress the response validity index on dummy variables for the samples and mobile phone use, among those who passed the front-end attention checks. This yields a less strong (negative) relationship (β = −0.10, P < 0.001, b = −0.04, 95% CI −0.04 to −0.03) relative to the full sample, or constraining to those who did not pass the attention checks (β = −0.12, P < 0.001, b = −0.07, 95% CI −0.1 to −0.04). Thus, the attention filters help, but do not fully resolve, the lower response validity on mobile phones. Overall, mobile phone access improves representativeness as it probably extends a platform’s reach. Yet, it is also associated with lower response validity (in the more representative samples), which can be partially but not fully addressed with attention checks. Importantly, one can typically restrict a sample’s mobile phone usage, which may be worth considering when the goal is a highly attentive sample, regardless of representativeness. A question for future work is how allowing or disallowing mobile phone usage influences samples within a given platform. Platforms versus samples A looming question unanswered in our analyses thus far concerns the potential confound between the platforms and the samples. Do variations stem more from the platforms or the precise types of samples drawn? We address this question by directly comparing the two Prolific and the two Connect samples. Both platforms have a pure convenience sample and a quota sample. These comparisons show that the quota samples from both platforms are predictably more representative than the convenience versions (Figs. 2 and 3), and response validity is the same across the sample types within each platform (Fig. 1 and Table 2). It is the platform, and not the sampling approach within the platform, that seems to shape response validity. Indeed, mobile phone usage remains largely the same across samples within a given platform. Moreover, for all the samples, front-end filtering improves response validity with no loss in representativeness. This result suggests that, generally, one should specify quotas and use front-end attention checks to optimize data collection. Costs The cost per participant varies substantially across samples18. In Supplementary Section 19, we provide details about the pricing of each platform at the time of survey, approximating the cost for a 15-min study. We emphasize that pricing is inherently time dependent; thus, one should only cautiously generalize about differences and should not make inferences about acute amounts. With these caveats in mind, we find that Forthright’s minimum participant pay per hour exceeds that of Prolific and Connect, while Open M-Turk, Lucid, and CR Toolkit do not appear to have variable compensation (and their suggested rates do not exceed those of Prolific and Connect). The relatively strong performance of Forthright (when using two filters) thus comes at a financial cost. We also identify the cost per respondent, for individual platforms, based on varying levels of attentiveness (Supplementary Table 19-2 in Supplementary Section 19). To the best of our knowledge, all platforms allow a researcher to reject participants who fail attention filters in the first minute of a study, such as the initial two filters we used. Many platforms, however, charge for respondents who complete a non-trivial amount of the survey (for example, 50% or more). Consequently, a researcher must pay for a respondent if they include filters later in a study (thereby paying for respondents they may not use). We thus compute the cost for a respondent who passes the first two attention filters but not later ones. We find, for example, that for Forthright, requiring respondents to pass the two filters and all of the other attention checks would cost US$8.35 more than if respondents were allowed to fail two attention checks beyond the two initial filters (that is, the former scenario means that a researcher pays for many respondents that they do not use because of a failed attention check). The analogous cost change on Connect is US$3.72. This difference highlights the non-trivial but variable costs of filtering respondents beyond the initial two filters. Increasing the number of attention filters can not only alter the representativeness of the sample, as we showed above, but also increase costs. On the flip side, some studies require a costly, highly attentive (albeit less representative) sample. In addition, some platforms allow for a priori agreements to filter respondents on the basis of more than two attention checks, which would alter cost computations. Distinct from actual compensation is perceived compensation. Platforms differ in the amount and form of compensation individual respondents can receive. We asked individuals whether they think they are regularly compensated fairly for their work on a given platform (on a 1–6 scale´, with higher scores being more perceived fairness). Interestingly, we find that participants’ perceptions of fair compensation do not predict response validity (β = −0.004, P = 0.623, b = −0.0003 95% CI −0.002 to 0.001) (Supplementary Section 20). Moreover, participants on Lucid are compensated in varying ways, including money, website coupons, video game currency and even nothing. We asked respondents how they were compensated and find that, within Lucid, those who reported not being compensated do not differ in their response validity relative to others (β = 0.02, P = 0.315, b = 0.01, 95% CI −0.01 to 0.03, controlling for dummies that represent variation in positive (non-zero) compensation) (also see Supplementary Section 20). These results may reflect some intrinsic motivation to perform well regardless of payment, and thus also suggest differences in sample constituencies by compensation. Of course, they may also be specific to our survey design, which is generally not highly cognitively taxing. Discussion We have evaluated response validity, representativeness and professionalism across nine online samples and have four key takeaways. First, we find substantial variation across platforms and samples; it is thus very misleading to make general statements about non-probability opt-in samples. Second, we find a negative relationship between overall platform response validity and platform representativeness (averaging over both demographic and attitude variants), r = −0.66, P < 0.001 (excluding Open M-Turk; Supplementary Section 21), a negative relationship between representativeness and past experience with similar studies (logged) (r = −0.26, P < 0.001), and a positive relationship between response validity and past experience with similar studies (logged) (r = 0.54, P < 0.001). Third, filtering out inattentive participants improves response validity in the more representative samples, but heavy filtering may affect representatives on some platforms. For Forthright, this culling is achieved by merely filtering on two trivial attention checks at the study outset, without compromising representativeness or losing many participants. For Lucid, this approach substantially helps, but the sample does not reach response validity parity with the other highly valid samples, while notably dropping its representativeness at the higher filtering levels. Fourth, applying weights to the less representative samples does not lead to consistent improvements in agreement with the more representative samples in terms of heterogeneous treatment effects (Supplementry Section 17). In addition, such effects may be more common in studies that use social or political rather than cognitive content. Our study, of course, has various limitations. First, we explored weighting as is commonly done by weighting based on the entire sample. However, it is possible that our results would have differed if we had instead weighted solely on the basis of attentive respondents. Although such an approach is unusual, it could be justified if the goal is to generalize to an attentive population. Second, we focused on samples where we had substantial control over the data collection processes; yet, we still have little understanding of how, precisely, the different platforms recruited respondents when constructing each panel. For instance, we do not know how the quota samples ‘filled’ the quotas and whether such efforts led to the recruitment of less-than-ideal (for example, inattentive or otherwise atypical) respondents69. This concern speaks to the need for greater transparency in participant recruitment70). Third, we used closed-ended conventional attention checks that spanned the length of the study to obtain a range of variation; however, attention checks should always precede experimental stimuli22,30. Further, in contrast to close-ended attention check, other approaches have been tested; for example, Kane and Barabas71 have pioneered an approach of using pretreatment checks that maximize attention to precise treatments12. A relevant question is whether such exacting checks, or alternative types such as those that rely on open-ended responses19, have any implications for representativeness. Fourth, these platforms are dynamic, changing their structures and samples over time. Our data came from a relatively constrained timepoint, and it is plausible that the timing impacts our results17,72,73. This speaks to the need to consider the interaction between samples and time. Fifth, our probability sample comparison was taken at a different timepoint and did not always have the exact same wording. Future work should collect probability responses concordantly and with the identical survey. Finally, while we explored nine samples, there are many more sample providers that could generate different patterns of results. Regardless, our findings highlight the need for researchers to carefully consider what study characteristics matter most in choosing a given sample. For example, high response validity may be particularly important for studies with complicated designs, or treatments/measures that require participants to invest substantial effort or attention. Representativeness may be particularly important for studies where individual correlations or heterogeneous treatment effects are of interest, especially if the study involves social or political content. We suggest the following guidelines.Researchers asking questions with low social/political content and with high attention requirements should consider one of the samples that scores higher on response validity and lower on representativeness, and that has more professional participants (for example, Connect, Prolific or CR Toolkit), with the addition of front-end attention filters.Researchers asking questions with social or political content and with low attention requirements (for example, shorter treatments or simpler measures) should consider one of the samples that scores higher on representativeness and that also has fewer professional respondents (for example, Forthright, Lucid or, to a lesser degree, Connect NR), with the addition of front-end attention filters.When representativeness is crucial, samples should not be constrained to participants using laptops or computers, as there is a strong association between mobile phone use and sample representativeness.Sample weights can help less representative samples recover experimental main effects but not necessarily heterogeneous effects.When collecting data and, critically, when using the more representative samples, researchers should always apply attention filters on the front end of their studies. Despite the variation in response validity, we were able to identify and remove inattentive participants from all samples using two simple attention checks embedded in the front-end demographics; thus, irrespective of sample, validity gains from filtering are always relevant.Two front-end attention filters do not undermine the representativeness of a sample.Most providers do not charge for respondents who fail attention checks placed at the start of a survey. This does not address all attention concerns but can help with overall response validity.Researchers asking questions involving social or political content and with high attention requirements as well as a need for participant naïveté should consider the sample that scores higher on representativeness and is still able to obtain highly valid data (that is, Forthright) with front-end attention filters. However, this option is more costly.There is no context in which researchers should be using Open M-Turk. It performed objectively worse than all others on all dimensions. These results, along with other recent work10, should clearly direct researchers away from this sample and to any of the other samples discussed. Researchers asking questions with low social/political content and with high attention requirements should consider one of the samples that scores higher on response validity and lower on representativeness, and that has more professional participants (for example, Connect, Prolific or CR Toolkit), with the addition of front-end attention filters. Researchers asking questions with low social/political content and with high attention requirements should consider one of the samples that scores higher on response validity and lower on representativeness, and that has more professional participants (for example, Connect, Prolific or CR Toolkit), with the addition of front-end attention filters. Researchers asking questions with social or political content and with low attention requirements (for example, shorter treatments or simpler measures) should consider one of the samples that scores higher on representativeness and that also has fewer professional respondents (for example, Forthright, Lucid or, to a lesser degree, Connect NR), with the addition of front-end attention filters.When representativeness is crucial, samples should not be constrained to participants using laptops or computers, as there is a strong association between mobile phone use and sample representativeness.Sample weights can help less representative samples recover experimental main effects but not necessarily heterogeneous effects. Researchers asking questions with social or political content and with low attention requirements (for example, shorter treatments or simpler measures) should consider one of the samples that scores higher on representativeness and that also has fewer professional respondents (for example, Forthright, Lucid or, to a lesser degree, Connect NR), with the addition of front-end attention filters.When representativeness is crucial, samples should not be constrained to participants using laptops or computers, as there is a strong association between mobile phone use and sample representativeness.Sample weights can help less representative samples recover experimental main effects but not necessarily heterogeneous effects. When representativeness is crucial, samples should not be constrained to participants using laptops or computers, as there is a strong association between mobile phone use and sample representativeness. When representativeness is crucial, samples should not be constrained to participants using laptops or computers, as there is a strong association between mobile phone use and sample representativeness. Sample weights can help less representative samples recover experimental main effects but not necessarily heterogeneous effects. Sample weights can help less representative samples recover experimental main effects but not necessarily heterogeneous effects. When collecting data and, critically, when using the more representative samples, researchers should always apply attention filters on the front end of their studies. Despite the variation in response validity, we were able to identify and remove inattentive participants from all samples using two simple attention checks embedded in the front-end demographics; thus, irrespective of sample, validity gains from filtering are always relevant.Two front-end attention filters do not undermine the representativeness of a sample.Most providers do not charge for respondents who fail attention checks placed at the start of a survey. This does not address all attention concerns but can help with overall response validity. When collecting data and, critically, when using the more representative samples, researchers should always apply attention filters on the front end of their studies. Despite the variation in response validity, we were able to identify and remove inattentive participants from all samples using two simple attention checks embedded in the front-end demographics; thus, irrespective of sample, validity gains from filtering are always relevant.Two front-end attention filters do not undermine the representativeness of a sample.Most providers do not charge for respondents who fail attention checks placed at the start of a survey. This does not address all attention concerns but can help with overall response validity. Two front-end attention filters do not undermine the representativeness of a sample. Two front-end attention filters do not undermine the representativeness of a sample. Most providers do not charge for respondents who fail attention checks placed at the start of a survey. This does not address all attention concerns but can help with overall response validity. Most providers do not charge for respondents who fail attention checks placed at the start of a survey. This does not address all attention concerns but can help with overall response validity. Researchers asking questions involving social or political content and with high attention requirements as well as a need for participant naïveté should consider the sample that scores higher on representativeness and is still able to obtain highly valid data (that is, Forthright) with front-end attention filters. However, this option is more costly. Researchers asking questions involving social or political content and with high attention requirements as well as a need for participant naïveté should consider the sample that scores higher on representativeness and is still able to obtain highly valid data (that is, Forthright) with front-end attention filters. However, this option is more costly. There is no context in which researchers should be using Open M-Turk. It performed objectively worse than all others on all dimensions. These results, along with other recent work10, should clearly direct researchers away from this sample and to any of the other samples discussed. There is no context in which researchers should be using Open M-Turk. It performed objectively worse than all others on all dimensions. These results, along with other recent work10, should clearly direct researchers away from this sample and to any of the other samples discussed. The choice of what data source to use can fundamentally shape what one finds, and ultimately the insights that inform research and policy decisions. Which data are best depends on the purpose of the research and resource constraints; our results provide guidance on how to choose and evaluate data sources when it comes to opt-in non-probability samples18. We hope that our approach provides a foundation for ongoing evaluations of opt-in non-probability samples. This is particularly vital given these samples can (and often do) evolve in terms of coverage and selection biases and are often not fully transparent in their construction. Considering the role of these data in understanding human behaviour, this should be a priority. Methods Ethics and reproducibility The study was approved by the institutional review board at the Massachusetts Institute of Technology complying with all relevant ethical obligations (protocols E-2443, E-4195 and E-5194). No deception was used; all participants provided informed consent and were compensated for their participation. Given the exploratory nature of this research, this study was not preregistered. All statistical tests are two-tailed. Samples For each of our samples (Table 1), we sought to use the standard sample features (for example, quotas and payment) implemented by the platform at the time of fielding (these standards may change over time). We identified the ‘standard’ via conversations with representatives from the platform or from information posted on the platform’s website. This ensures capturing what a ‘typical’ researcher could expect from using the sample. Consequently, we utilize the ‘standard’ quotas, even though researchers could work with many of the platforms and pay more to institute distinct (personalized) quota options. In two cases, the platforms offer more than one ‘standard’ option. Connect and Prolific both provide two sample options that come from the same platform but vary in the number of quotas offered as the default. We use the labels ‘Connect’ and ‘Prolific’ to indicate the options with fewer quotas and ‘Connect NR’ and ‘Prolific NR’ for the options that incorporate more quotas (where NR indicates ‘nationally representative’). We exploit this variation within platforms to evaluate whether platform oversight matters across sample types. Otherwise, our samples include substantial quota variance, ranging from the many quotas with Forthright and Lucid to none with CR Toolkit, Connect, Open M-Turk and CRSTAL (Table 1). We recognize that comparing quota versus non-quota, non-weighted samples introduces differences in sample composition. This motivates our analyses of sample weights. The samples also reflect variation in panel maintenance. For instance, we believe Open M-Turk exercises minimal to no oversight, whereas Forthright, managed by Bovitz Inc., procures and maintains its panel. CRSTAL is an example of an ‘in-house panel’ managed by a private social science laboratory (the Polarization and Social Change Lab at Stanford University). We aimed to recruit N = 1,000 participants per sample, except for Lucid, for which we aimed to recruit N = 4,000 participants owing to known high inattention and attrition rates. Specific details for each sample, including data usage, can be found in Supplementary Section 1. Sample sizes were chosen via a combination of maximizing power while also working within budgetary constraints. Recruitment procedure Data were collected between June 2022 and October 2023. The overall study duration had a median time of 18.3 min for those who completed the full study. In total, 13,053 participants started the study. A total of 260 individuals (for whom we could track) participated in the study through multiple samples, which is not surprising given that platforms have a limited number of active participants74. We included all their responses because, in most cases, researchers do not know (unless asked) about other studies or platforms in which a respondent is participating. Reporting summary Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article. Supplementary information Supplementary InformationSupplementary Sections 1–28, including Supplementary Tables 1–12 and Supplementary Figs. 1–95. Reporting Summary Peer Review File Supplementary Sections 1–28, including Supplementary Tables 1–12 and Supplementary Figs. 1–95. Reporting Summary Peer Review File