The effect of manuscript characteristics on quality assessment of scientific manuscripts in psychology Authors: Hilde E.M. Augusteijn, Jelte M. Wicherts, Klaas Sijtsma, Marcel A.L.M. van Assen, Sonia Vasconcelos, Sonia Vasconcelos, Sonia Vasconcelos, Sonia Vasconcelos, Sonia Vasconcelos Published: 2026 ABSTRACT We report a vignette study and a survey to investigate which study characteristics influence quality ratings psychology academics give of articles submitted for publication, and psychology academics and students to theses written by psychology students. In the vignette study, 800 respondents evaluated the quality of an abstract (7-point scale) of studies with small (N = 54) or large sample sizes (N = 540), showing statistically significant or non-significant results, and containing statistical reporting errors or no errors. In the subsequent survey, the same participants rated the importance of 29 manuscript characteristics related to a study’s theory, design, conduct, data analyses, and presentation for assessing either the quality of a manuscript or its publishability (article) or grade (thesis). Results showed that quality ratings were affected by sample sizes but not by statistical significance or the presence of statistical reporting errors in the rated research vignette. Furthermore, in the survey academics and students provided highly similar ratings of the importance of different aspects relevant to quality assessment of articles and theses. These results suggest that quality criteria for scientific manuscripts are already adopted by students and are similar for submitted manuscripts and theses. Further research is needed to examine which study characteristics researchers consider most important when assessing a manuscript’s quality in peer review. Introduction Publication bias is widespread in many research areas. In psychology, over 90% of the published papers report a statistically significant effect [1–3], but the average power of studies is estimated well below .50 [4–7]. This high prevalence of statistically significant results is inconsistent with the small effects that are often studied using underpowered designs [8]. The sources of publication bias are twofold. On the one hand, journals generally prefer publishing articles with statistically significant results [9,10]. Journal editors prefer articles with nice stories based on novel and statistically significant results and tend to ignore articles with unexpected null results and many ifs and buts [11]. On the other hand, authors themselves cause publication bias [12–14]; they are less likely to submit results that are not statistically significant. Perhaps authors already anticipate possible rejection or lose interest in their project when their expectations are not met. Either way, studies with statistically non-significant results are more likely to end up in the so-called file drawer, producing a biased representation of a research topic or area based on only the significant results [15]. To avoid the file-drawer effect and resulting publication bias, authors have an incentive to increase the probability of finding a statistically significant result. They can increase the probability while designing their study but also after data collection has been completed. The ‘correct’ way of improving one’s chances if a true nonzero effect exists is to increase statistical power using larger samples or by using appropriate methodology (e.g., meta-analysis [16]) or statistical techniques [17]. However, due to practical or financial constraints, it is often difficult for researchers to increase their sample sizes. Another, albeit questionable way to increase the probability of finding a statistically significant result, is the opportunistic use of so-called researcher degrees of freedom [18]. Researchers can use these degrees of freedom to increase type I error [19] and statistical power by analyzing their data in various ways, decide to stop data collection after looking at the results, drop conditions from analyses, or remove outliers. These manipulations are also known as p-hacking, which is an example of questionable research practices (QRPs). Unfortunately, many psychology researchers have admitted having engaged in QRPs [20–24]. QRPs and publication bias are considered important causes of irreproducible findings in psychological research [25]. Peer review is a gatekeeper that is expected to protect scientific literature against publication bias and QRPs and to guarantee that the published scientific studies meet a minimum level of quality. Given that much of the published literature does not seem to meet the high-quality criteria we desire, and published results often fail to replicate [(e.g., 25, 26)], the current study examines which manuscript characteristics influence peer reviewers’ quality assessment of scientific manuscripts, including students’ master’s theses. We also examine which manuscript characteristics are assumed to be important for quality assessment by scientists and students. We first discuss what is currently known about quality assessment in peer-review and education before presenting our research questions and hypotheses. Peer review Following Kelly et al. [27, p. 227], we define peer review as “a process of subjecting an author’s scholarly work, research or ideas to the scrutiny of others who are experts in the same field”. The peer review process has been a formal part of scientific communication since the first scientific journals were published over 300 years ago [28]. Although there are differences between journals, peer review typically entails the next steps: (a) an editor makes an initial manuscript selection by checking journal fit and whether a submitted manuscript meets the formal standards of, for example, format and ethics; (b) the editor selects and invites reviewers; (c) the reviewers independently assess the quality of the work and make recommendations concerning publication; and (d) if the editor decides the manuscript is a candidate for publication, she guides the process of resubmission and additional rounds of review and decides upon definitive publication. The first goal of peer review is to check whether reported research meets the appropriate standards, and therefore whether findings and conclusions can be considered valid. The second goal is to help authors improve the quality of their research and its presentation. The third goal is to assess originality, significance, and broader interest, and finally, to assess the ‘fit’ between a paper and a journal [29]. Despite its ubiquitous use, peer review is intensely debated. Ware [30] found that most academics (85%) believe that peer review greatly helps scientific communication. In contrast, others argued that peer review might increase the use of QRPs due to reviewers demanding ‘perfect’ results that are unrealistic [11,31], and/or researchers expecting reviewers to demand overly clean results [24]. It may well be that critical peer reviewers stimulate, either implicitly or perhaps sometimes even explicitly, authors to leave out unconvincing outcomes or aspects of the study [22,24] or even to misreport certain results. Hence, the quality of reviews remains a concern and little consensus exists on how to even define review quality [32,33]. It is unclear to what extent peer review helps improve research manuscripts. Peer review seems to bring about little change in manuscripts from preprint to post-print [32], which does not need to imply that the original research was flawless. Fraud and QRPs still enter the published literature, and published articles are retracted from publication after peer review failed to detect problems in these articles [27,32,33]. Furthermore, experimental studies have shown that reviewers have a hard time detecting even obvious errors in a paper [34–36]. Although most authors and reviewers believe that the detection of plagiarism is a task of peer review, a minority believes peer review is suited to do so [27]. Reviewers are often inconsistent and agree only slightly above chance in their opinion on whether a paper should be published [37,38]. Inter-reviewer reliability is low [39,40], with quality ratings of the same manuscript ranging from unacceptable to excellent [33,41]. Whereas one reviewer argues ‘I found this paper an extremely muddled paper with a large number of deficits’, another reviewer argues ‘It is written in a clear style and would be understood by any reader’ [38]. Assuming almost all reviewers are honest and are not abusing their power of peer review (e.g., to stall publication of competitors or scoop their ideas), these inconsistencies indicate that the evaluation of manuscript quality is ambiguous. Every peer reviewer has a different background and different knowledge, and editors might be looking for peer reviewers with differing expertise to review an article. While this is a sound strategy using different reviewers’ complementary expertise, unanticipated differences in knowledge and understanding of the same topic, knowledge of previous studies on the same topic, methodological and statistical skills, and personal preferences for some topics but not for others may readily produce inconsistent review assessments. Differences between reviewers in knowledge, understanding, and preferences are difficult to control in research. Therefore, in this study, we focus on the question of which manuscript characteristics are most important for manuscript quality rating that might differ between reviewers and even between manuscripts. Previous research indicates that reviewers seem to favor their judgements of scientific manuscripts on ‘false cues’, such as statistical significance, often unduly complex procedures, and overly complex writing [42]. Furthermore, reviewers suffer from the same biases as researchers, such as confirmation bias and hind-sight bias [43,44]. They might not even realize these false cues influence their quality assessments. For example, in an experimental study by Atkinson, et al. [45], peer reviewers received a version of a manuscript with either statistically significant results (p < .01), results with p < .10, or results with p < .25. The manuscript that reported a statistically significant result was three times less likely to be rejected than the other two versions. Despite all other characteristics of the manuscript being equal, reviewers often believed they rejected the manuscript based on the design of the study rather than its results. It is unclear if reviewers differ in their assessment of the quality of a manuscript or whether they assess the suitability of the manuscript for publication. Due to (perceived) expectations of journals and editors, perhaps other characteristics are important for assessing publishability of an article. Statistical significance may be considered more relevant for publishability than for manuscript quality. Reviewers may recognize high quality research but use different characteristics when assessing its ‘publishability’. Review in education There are other review issues relevant in assessing a master’s thesis than in reviewing scientific manuscripts as part of the publication process. With their master’s thesis, students show what they have learned during their education. We would expect teachers to positively consider cues that indicate responsible research practices (RRPs), but to negatively assess indications of QRPs. Since the main goal of a master’s thesis is educational, that is, the application of knowledge, insights, and skills previously learned, but usually not publication, statistical significance of the results should be less important than in manuscripts submitted for publication. In addition, students do not believe that there is a causal relation between good science and statistically significant results, and they do not believe their teachers reward statistically significant results [46]. If students’ perspective is correct, we expect academics to uphold different quality standards for master’s theses than for research manuscripts. Furthermore, theses differ from published manuscripts in many aspects that may be relevant for their review and evaluation. For example, previous research showed that theses report effect sizes more frequently than published manuscripts [46–48], on average use larger samples [49], and conduct a priori power analysis more often [46,50]. These differences could also indicate that students and researchers consider different aspects in assessing the quality of these manuscripts. Students are often confronted with published literature as examples of good research reports during their education. These examples, as well as students’ perceived attitude of their academic teachers towards QRPs shape the students’ own attitudes and behavior towards these QRPs [46] and set the bar for what is considered high-quality research. For example, what students believe may be a sufficient sample size (e.g., to have sufficient statistical power) might be influenced by sample sizes encountered in the literature studied in various courses, and when studying in a field where open science is advocated in the published literature, students might assign greater importance to pre-registration and sharing study materials. Previous research shows that both researchers and students frequently engage in QRPs. Over 90% of psychology researchers admitted having engaged in QRPs during their career [20,21,23,51]. Estimates for students indicate that 64% of them have already engaged in QRPs in their short academic careers (up to graduate level, [52]), whereas 40% of psychology students engaged in QRPs in their thesis [46]. It is currently unknown how academics differ in their quality assessment of theses and articles, and how potential QRPs affect this assessment. Finally, it is unknown how students perceive this quality assessment of their supervisors. Current study We know little about the characteristics of submitted manuscripts and master’s theses that affect quality assessments and their publishability or grades. It also is unknown whether students and their supervisors believe quality is affected by the same characteristics. Therefore, the main research question of this study is “Which characteristics of a manuscript affect students and academic researchers when assessing the quality of a scientific manuscript?”. We note that the preregistered research question was originally formulated as: “Which characteristics of a manuscript do students and researcher believe to be of importance when assessing the quality of a scientific manuscript?”. This formulation was changed, because we believed, in hindsight, that this formulation did not cover the full range of our study. We did not only investigate the beliefs of academics and students, but also tested what influenced their quality assessment in the vignette study that we preregistered together with the survey study. In our vignette study, we focused on three research characteristics that we manipulated in a study’s abstract to examine their effect on the quality evaluation of scientific manuscripts (article or thesis). We studied the impact of sample size, statistical significance, and a statistical reporting error on quality ratings of the research. A statistical reporting error refers to an inconsistency between the reported test statistic, degrees of freedom, and p-value. Furthermore, we studied the two-way interactions of these three characteristics, and the effect of career phase (student versus academic) and manuscript type (thesis versus article) on the ratings of the research. The following research questions and matching hypotheses were formulated and preregistered (https://osf.io/cd3uw): RQ1: Which characteristics of research (sample size, significance, reporting error) as described in a manuscript (thesis or article) affect the quality assessment of a scientific manuscript by both students and academics? H1a: Manuscripts with large sample size are rated as having higher quality compared to manuscripts with smaller sample sizes H1b: Manuscripts with significant results are rated as having higher quality compared to manuscripts with non-significant results H1c: Manuscripts with reporting errors are rated the same as manuscripts without reporting errors RQ2: Do the effects of the characteristics on the quality of the manuscript as assessed by academics differ between a thesis and an article? H2a: The differences in quality rating between small and large sample size manuscripts is larger for articles compared to theses H2b: The differences in quality rating between significant and non-significant manuscripts is larger for articles compared to theses H2c: The differences in quality rating between manuscript with and without reporting error do not differ for articles compared to theses RQ3: Do the effects of the characteristics on the quality of a thesis differ between students and academics? H3a: The differences in quality rating between small and large sample size manuscripts differs between students and academics H3b: The differences in quality rating between significant and non-significant manuscripts is larger for students compared to academics H3c: The differences in quality rating between manuscripts with and without reporting error does not differ between students and academics We explored the importance of a broader set of manuscript characteristics that could be relevant for the evaluation of manuscript quality. We formulated no a priori hypotheses for these characteristics but considered a set of research questions to compare different conditions. First, we studied which characteristics were considered relevant when assessing the quality and the grading of a thesis (RQ4) and when assessing the quality and the publishability of an article (RQ5). For academics, we studied the differences in the relevance of the different characteristics when assessing quality and when assessing publishability/grade for both theses (RQ6) and articles (RQ7), differences between the importance of characteristics when assessing quality of theses versus articles (RQ8), differences between the importance of characteristics when assessing the grade or publishability of theses versus articles (RQ9), and differences between the differences of the importance of characteristics when assessing quality versus grade/publishability of theses versus articles (RQ10). For students, we studied differences in the relevance they believe their supervisors assign to characteristics when assessing quality versus grade (RQ11). Finally, we studied differences in relevance of characteristics between students and academics when assessing quality of thesis (RQ12) and grades (RQ13). Fig 1 shows research questions 6–13 and their relations. Overview of Explorative Research Questions of the Survey Study on the Evaluation of Scientific Manuscripts. Six different evaluations are made. Academics (top) evaluate both ‘quality’ and ‘grading’ of theses (left) and the ‘quality’ and ‘publishability’ of articles (right). Students (bottom) only evaluate the ‘quality’ and ‘grading’ of’theses’ (left). Arrows 6-9 and 11-13 refer to correlations between two evaluations. For instance, and more specifically, arrow 9 corresponds to the correlation between the evaluated importance of 29 manuscript characteristics for ‘grading theses by academics’ and the evaluated importance of these same manuscript characteristics for the ‘publishability of articles by academics’. Arrow 10 refers to a difference between two correlations. The preregistration lists all research questions verbally. Method Design The pre-registered project (https://osf.io/cd3uw) consists of a vignette study and a survey. Conditions varied across six design factors. First, participants were either students or academics (factor: career phase). Second, participants were asked about the evaluation of either theses (academics and students) or articles submitted for publication (academics only) (factor: manuscript type). The third, fourth and fifth factors pertained to only the vignette part of the study. The text of the vignette was an abstract describing a study with either a small or a large sample size, main results were either statistically significant or not, and the vignette either contained a statistical reporting error or not. The sixth factor only pertained to the survey part of the study. Participants of the vignette were subsequently asked in the survey to evaluate the importance of a list of manuscript characteristics related to either (i) quality assessment of an article/thesis, or (ii) publishability of an article or a thesis grading (factor: quality assessment type). In total, there were 48 conditions (for academics: 2 manuscript type × 2 sample size × 2 statistical significance × 2 reporting error × 2 quality assessment type = 32. For students: 2 sample size × 2 statistical significance × 2 reporting error × 2 quality assessment type = 16). Power analysis To determine the sample size needed to evaluate the hypotheses, we conducted an a priori power analysis. Since we only formulated hypotheses for research questions 1–3, we based our power analysis on these hypotheses. Based on a fixed-effect ANOVA with 24 conditions and the effect of a factor with two levels (df = 1), desiring a power of .8 with α = .05, and assuming a small effect size f = .10, we needed at least 787 participants. Assuming 33 participants per condition, this meant 528 academics and 264 students. Participants Data were collected among psychology students and academics who had published within the field of psychology. Exclusion criteria for students were not studying psychology or having finished one’s education program. For academics, exclusion criteria were having no experience with grading theses (in thesis condition) or having no experience with peer reviewing an article (in article condition). PhD candidates were considered as academics rather than students. The sampling strategy between academics and students differed substantially, and we therefore describe their data collection procedures separately. In November 2019, contact information of academics who published articles or editorial materials in psychology was extracted from Web of Science. E-mail addresses were automatically matched to the author names and academics received a personalized invitation to complete the survey. Three rounds of sampling were needed to generate enough responses. Data collection took place between February 2020 and April 2020. In these three rounds, a total of 11,555 authors and editors was contacted, and 1,239 of them started the survey (response rate of 10.7%). However, many of them did not complete the survey (and were removed from the dataset) or were excluded from the sample due to the exclusion criteria. A total of 687 academics provided responses that were included in the analyses (response rate 5.9%). Fig 2 provides an overview of the sampling procedure of academics. More detailed information can be found on OSF (https://osf.io/j2gpk/). Participants were randomly assigned to one of the 32 academics conditions. Overview of the Sampling Procedure Amongst Academics. The student survey was distributed via social media, as well as through the professional network of the primary researchers and their direct colleagues. An invitation to participate was placed on Twitter, as well as 51 international psychology student Facebook groups. Unfortunately, Facebook flagged the invitation as spam and removed all messages. In a second attempt, moderators from 49 large psychology student Facebook groups were asked to post a message with the survey invitation in their Facebook groups. 24 groups replied positively and either posted the message themselves or allowed us to post the message and pinned it to the top of the Facebook group so it would be noticeable. These Facebook groups had almost 65,000 members in total. The survey was also posted on two survey sharing Facebook groups, with an additional 40,000 international psychology students. Despite the effort, response rate was very low amongst international psychology students. After 11 months, data collection was closed after 301 students participated in the survey. Participants were randomly allocated to the 16 student-conditions. Unfortunately, 188 of the 301 participants needed to be excluded from analysis, because they did not complete the survey (160), did not study psychology [20], or had recently graduated from their studies or worked already as a PhD candidate [8]. The final number of student responses included was 113. Due to both the low number of responses and the subsequent low statistical power to detect small effect sizes, all hypotheses and analyses that include students (Research questions 3, 11, 12 and 13, see Fig 1) should be considered exploratory instead of confirmatory. Participant characteristics Researchers and students included in our sample originated from all continents. Academics were in different stages of their career and worked in various research fields (see Table 1 for participant characteristics). Many researchers (256, 37.2%) indicated they did not work in any of the pre-specified research areas, even though these participants had published on topics within the pre-specified area of psychology. Deviating from our preregistration, we decided not to remove participants from other fields, thereby retaining the planned statistical power for our analyses (note: tests of hypotheses H1a-c to H3a-c excluding those participants yielded the same substantive conclusions, see https://osf.io/84ugb). 10.1371/journal.pone.0355938.t001Table 1Characteristics of Participating Researchers and Students.Researchers(n = 687)Students(n = 113)ContinentEurope338 (49.2%)Europe, the Netherlands55 (49.1%)Europe, other20 (17.9%)North America222 (32.3%)8 (7.1%)South America35 (5.1%)3 (2.7%)Africa9 (1.3%)12 (10.7%)Asia47 (6.8%)11 (9.8%)Australia & Oceania36 (5.2%)3 (2.7%)Missing1 (0.9%)Career phasePhD77 (11.2%)PostDoc55 (8.0%)Researcher46 (6.7%)Teacher10 (1.5%)Assistant Professor100 (14.6%)Associate Professor157 (22.9%)Full Professor200 (29.1%)Other42 (6.1%)3 (2.7%)Bachelor73 (64.6%)Master37 (32.7%)Research FieldApplied Psychology74 (10.8%)Biological Psychology10 (1.5%)Clinical Psychology45 (6.6%)Developmental Psychology37 (5.4%)Educational Psychology49 (7.1%)Experimental Psychology52 (7.6%)Mathematical Psychology15 (2.2%)Multidisciplinary Psychology33 (4.8%)Psychoanalysis2 (0.3%)Social Psychology114 (16.6%)Other256 (37.3%) Characteristics of Participating Researchers and Students. Ethical review and data management This research project received an exemption for ethical review by the ethical review board of Tilburg University (ERB, EC-2019.EX135) due to anonymous data collection with minimal risk since the obtained information (a) cannot be traced directly or indirectly to the individual, and (b) does not harm or discomfort the individual in any way. Informed consents were also reviewed by the ERB. Informed consent was obtained by all participants at the start of the survey, by confirming to participate by clicking a button. The data management plan has been approved by the data representatives at Tilburg University. The data management plan describes data storage, open data plans, a pre-DPIA (Data protection Impact Assessment) and a description of GDPR agreements and compliance. Both are available from the preregistration on OSF (https://osf.io/cd3uw). Procedure All data were collected using Qualtrics. Participants first answered a set of general questions. Students were asked whether they studied psychology, where they studied (continent), in which phase of their study they were (bachelor/undergraduate, master/graduate, other, namely...), and whether they had ever written a thesis (bachelors’ and/or masters’ thesis). Academics in the thesis condition were asked if they had ever graded a thesis (bachelor/undergraduate level, master/graduate level, PhD level), and in the peer-reviewed manuscript condition they were asked whether they had ever peer reviewed an article submitted for publication. Furthermore, all academics were asked where they worked (continent), what position they held, and in which psychology research field they primarily worked. Categorization was based on the psychology subfields as distinguished in the Clarivate Analytics Master Journal List (https://mjl.clarivate.com). If participants indicated that they did not study psychology (student conditions), had never graded a thesis (academic-thesis condition), or had never peer reviewed a paper (academics-article condition), they were thanked for their participation and redirected to the end of the questionnaire. Vignette study Participants were asked to read an abstract of a manuscript that was submitted for publication (academics), or an abstract of a master’s thesis (students and academics). The text of the abstract was written by the principal investigators, and its face validity was assessed by a clinical psychologist in the field of personality psychology (see Box 1). Three characteristics of the abstract were manipulated; sample size (small or large; italics: N = 54 or N = 540), statistical significance of the main finding (yes or no; underlined: p = .005 or p = .6), and reporting error (yes or no; bold; p < .001 (incorrect) or p = .01 (correct)). Box 1. Text of Vignette Study with Manipulated Characteristics Sample size (Italics), Statistical Result (Bold), and Reporting Error (Underlined). Abstract Background: Test anxiety can have a large impact on a student’s academic career. Previous research shows higher anxiety, as well as higher neuroticism and conscientiousness, for women than for men. We examined whether sex differences in neuroticism and conscientiousness can explain possible sex differences in test anxiety. Methods: In this experimental study, [540/ 54] participants ([300/ 30] male and [240/ 24] female) filled out the Mowen’s Personality Scale and were randomly assigned to one of two conditions. In the high-stakes condition, participants watched a video clip and completed a fast-paced 10-item test about what they had seen (memory task); payment depended on recall performance, and participants knew they were graded relative to other participants. In the low-stakes (control) condition, participants watched the same video clip but answered ten unrelated questions; participants only received a show-up fee. All participants were asked to report anxiety (using the Spielberger State-Trait Anxiety Inventory) twice: once before randomization (pre-test) and again just before completing the ten items described above (post-test). Results: The increase in anxiety (i.e., post-test – pre-test) between the two conditions was, in line with previous research, higher for women than for men ([t(536) = 2.33, p < .001/ t(50) = 2.40, p < .001/ t(536) = 2.33, p = .01/ t(50) = 2.40, p = .01]). [Contrary to/ Confirming] our hypothesis, sex differences in neuroticism and conscientiousness could [not] explain the sex difference in increased anxiety ([F(4,532) = 0.689, p = .6/ F(4,46) = 0.694, p = .6/ F(4,532) = 3.76, p = 0.005/ F(4,46) = 4.28, p = .005]). Conclusions: Sex differences in test anxiety are [not/ at least partially] explained by sex differences in neuroticism and conscientiousness. Based on the abstract, participants were asked to rate the quality of the master’s thesis/scientific manuscript on a 7-point scale ranging from very low quality to very high quality. Next, they were asked to name three aspects of this thesis/manuscript that were most relevant for their assessment of its quality. Survey The questionnaire asked participants how important different manuscript characteristics are when assessing the (i) quality and (ii) grading of a thesis (by students and academics), and the (iii) quality and (iv) publishability of an article (only by academics). The instructions of each condition are provided in Table 2. 10.1371/journal.pone.0355938.t002Table 2Formulation of Survey Instruction for all six Conditions.Student condition –thesis QualityStudent Condition – thesis GradingResearcher Condition – thesis QualityResearcher Condition – thesis GradingResearcher condition – article QualityResearcher Condition – article Publishability“In the following questions, you will be asked to indicate the extent to which you believe different characteristics of the study/report are important when...supervisors assess the quality of a master’s thesis.”supervisors determine the grade of a master’s thesis.”assessing the quality of a master’s thesis.”determining the grade of a master’s thesis.”assessing the quality of a submitted manuscript.”assessing a manuscript’s suitability for publication in a peer-reviewed journal.” Formulation of Survey Instruction for all six Conditions. After these instructions, participants were asked to rate the importance of 29 manuscript characteristics (see Table 3) on a 7-point scale ranging from ‘Not at all important’ to ‘Of the utmost importance’. These 29 characteristics were related to different aspects of a manuscript: theory, design, conduct, (data) analysis, and presentation. All 29 characteristics were presented in a random order different for different participants. 10.1371/journal.pone.0355938.t003Table 3Manuscript Characteristics Evaluated by the Participating Researchers and Students.CategoryAbr.Manuscript characteristicTheoryT1Describing the relevant theories.T2Deriving hypotheses from theory.T3Explicitly stating the hypotheses to be tested.T4Using a large number of literature references.T5Clearly describing the study objective.DesignD1Using an appropriate study design.D2Clearly describing the sample population.D3Reporting pre-study calculation of sample size.D4Having a large sample size.D5Using a complex design.D6Pre-registering (part of) the study.ConductC1Achieving a high response rate.C2Achieving high statistical power.C3Applying complex statistical analyses.C4Providing open data (i.e., making data available online).C5Providing open research materials (i.e., making the materials available online or in the manuscript).C6Providing open analysis code (i.e., making the code available online or in the manuscript).Analysis & presentationA1Adequately describing all used statistical procedures.A2Using the appropriate statistical analyses.A3Distinguishing confirmatory from exploratory statistical analyses.A4Reporting assumption checks.A5Clearly presenting the statistical results.A6Providing confidence intervals for the main statistical results.A7Providing effect sizes for the main statistical results.A8Reporting the statistical results without errors.A9Observing large effect sizes.A10Observing the main effect in the hypothesized direction.A11Reaching statistical significance of the results of the main hypothesis.A12Drawing the correct conclusion from the statistical analysis. Manuscript Characteristics Evaluated by the Participating Researchers and Students. Statistical analysis Data were analyzed using R (version 4.1.1). In the preregistration, an analysis plan was described, and R-code was included to answer research questions 1–3, as well as R-code for a planned principal component analysis (PCA) that could be used to prepare data for research questions 4–13 (https://osf.io/fn5tu/). Research questions (RQ) 1–3 were answered using linear regression analyses, by testing the main effects of manuscript characteristics (sample size (H1a), statistical significance of the effect (H1b), reporting error (H1c)); by adding to the model and testing the three interactions manuscript characteristic × manuscript type (thesis vs. article) (H2a-H2c); and by adding and testing the three interactions manuscript characteristic × actor (student vs. academic (H3a-H3c) (see preregistration for the equations). As the outcome variables are Likert scales with seven ordered response categories, we also analyzed the data using quantile (median) regression (Quantreg R-package, version 5.86, [53]) as a sensitivity analysis. For the confirmative analyses (vignette study, hypothesis 1–3), the alpha level was set to.05. Since hypotheses 1c, 2c and 3c indicated no effect and null hypothesis significance testing cannot provide support for null hypotheses we evaluated these hypotheses using Bayesian statistics. We computed posterior model probabilities for four models (L(data|μi)), assuming a zero (μ0), small (μs), medium (μm), or large effect (μl), respectively, all having the same prior probability (uniform prior distribution). The relative posterior model probabilities of the effect size (zero, small, medium, large) were computed as with L being the likelihood of the observed F-value, given a zero, small, medium or large effect, using the (non-central) F-distribution. The observed F-value is the F-statistic of the test comparing the regression model with the parameter to be estimated (a simple effect of ‘reporting error’, or the main effect in case the interaction was not statistically significant) to the regression model without that parameter. L is distributed as F(1,N-k-1,λi), with k the number of predictors in the model (k = 3 for H1c, k = 6 if the interaction is significant, k = 5 if the interaction is not included in the model for H2c and H3c), and non-centrality parameter λi = 0, 0.02×(N-k-1), 0.15×(N-k-1), 0.35×(N-k-1) for a zero, small, medium, large effect, respectively (see OSF, https://osf.io/sk5t7/, for R-code). We interpreted a relative posterior probability of at least .75 for a zero true effect-size (corresponding to a Bayes Factor of 3 or more) as evidence in favor of the null-hypothesis (i.e., a zero true effect-size). The aspects the respondents named in the open-ended question that followed the vignette study (“Please name the three aspects of this manuscript that were most relevant for your assessment of its quality”) were analyzed using content analysis. The aspects named by participants were categorized by the principal investigator (HA). Categories emerged from the data itself, and specific categories with less than five mentions were put in the ‘other category’. To analyze the importance of the 29 manuscript characteristics in the survey, a PCA was planned for each of the six survey conditions separately to reduce the dimensionality of the dataset if (i) the Kaiser–Meyer–Olkin (KMO) test indicated sufficiently correlated items (KMO > .6 in all six conditions) and at the same time (ii) items loaded similarly on the principal components across the six conditions. If either of these two criteria were not met, analyses would take place at the item-level and the four intended categories (theory, design, conduct and analysis, and presentation). More details can be found in the preregistration. The statistical analysis of research questions 4–13 was exploratory and was not part of the preregistration. To investigate these research questions, we provide descriptive statistics and correlations between the importance ratings of either the new principal components (RQ 4–5), or their four intended categories (Table 3), of the conditions compared in research questions 6–13 (see also Fig 1). For these exploratory analyses, as a (partial) correction for multiple testing, the alpha level was equal to .01 instead of .05. This provides us with insight in the characteristics most relevant in the quality assessment, according to each group (academics vs students, articles vs theses, quality vs grading/publishability), and the correlations inform us to what extent these groups (dis)agree in their importance ratings. Both the data and the codebook of the data can be found at osf.io/j2gpk. Results Vignette study The mean quality rating of the research manuscript abstract (Box 1) for all participants was 3.84 (sd=1.42). Students on average provided higher quality ratings than academics (4.42 versus 3.75, t(156.02)=4.92, p<.001, d=0.499), and academics on average provided higher quality ratings to the thesis abstracts than to the scientific manuscript abstracts (3.96 versus 3.55, t(678.01)=3.83, p<.001, d=0.292). Table 4 provides mean quality ratings, standard deviations, number of participants in each of the 18 conditions of the vignette study, and effect sizes (Cohen’s d) of differences between two conditions. 10.1371/journal.pone.0355938.t004Table 4Mean Quality Ratings of Scientific Manuscripts for all Conditions and Effect Sizes Comparing Conditions.Students (113)Academics (687)ComparisonThesis (113)Thesis (344)Article (343)Thesis:Stud-academ.Academics: thesis-articleSample sizeSmall4.31 (1.40, 62)3.82 (1.47, 166)3.43 (1.31, 171)d = 0.343(p = .023)d = 0.281(p = .010)Large4.57 (1.25, 51)4.08 (1.45, 178)3.67 (1.32, 172)d = 0.373(p = .021)_d = 0.299(p = .005)d = 0.199d = 0.181d = 0.183SignificantNonsig4.55 (1.12, 71)4.01 (1.48, 173)3.54 (1.28, 169)d = 0.440(p = .002)d = 0.334(p = .002)Sig4.21 (1.63, 42)3.91 (1.46, 171)3.55 (1.36, 174)d = 0.193(p = .268)d = 0.251(p = .020)d = − 0.229d = − 0.068d = 0.006ReportingPresent4.52 (1.37, 54)3.95 (1.50, 169)3.53 (1.30, 170)d = 0.408(p = .011)d = 0.298(p = .006)ErrorAbsent4.34 (1.31, 59)3.97 (1.44, 175)3.57 (1.35, 173)d = 0.278(p = .068)d = 0.286(p = .008)d = −0.134(p = .479)d = 0.013(p = .905)d = 0.028(p = .795)Note. Standard deviations and sample sizes for each condition between brackets. The penultimate column shows effect sizes of the difference between the quality rating of the thesis abstract by students and academics. The final column shows effect size of the difference between thesis and article abstracts by academics. Mean Quality Ratings of Scientific Manuscripts for all Conditions and Effect Sizes Comparing Conditions. Note. Standard deviations and sample sizes for each condition between brackets. The penultimate column shows effect sizes of the difference between the quality rating of the thesis abstract by students and academics. The final column shows effect size of the difference between thesis and article abstracts by academics. Below each design factor (sample size, significance, and reporting error) in Table 4, the effect size of the design factor within a condition is provided. None of these factors influenced the quality ratings within a group of respondents and had generally small effect sizes. Effect sizes for small versus large sample size manuscripts differed from d=0.181 to d=0.199. The effect sizes for manuscripts with statistically significant versus non-significant results differed from d=−0.006 through d=0.228. The presence versus absence of a reporting error resulted in effect sizes between d=−0.028 and d=0.134. We conducted a linear regression analysis using all respondents (academics and students) to answer the three research questions of hypothesis 1. Sample size, statistical significance, and reporting error were entered as predictor variables to study their impact on the quality rating of the abstract. Only sample size (H1a) predicted quality rating (B=−0.25, SE=0.10, 95% CI:[−0.44, −0.05], t=−2.46, p=.014), suggesting that small sample sizes received lower quality ratings. Statistical significance (H1b; (B=0.14, SE=0.10, 95% CI:[−0.06, 0.34], t=1.39, p=.166) and reporting errors (H1c; (B=0.002, SE=0.10, 95% CI:[−0.19, 0.20], t=0.02, p=.983) did not predict quality rating. The omnibus test, however, indicated that the model with these three predictors together did not predict quality rating, with only 1% of the variance of quality rating explained (95% CI:[.00, .02],F(3, 796)=2.57, p=.053). Exploratively, we also tested the sample size × statistical significance interaction, but found no effect either (B = 0.17, t = 0.85, p = .398). For hypothesis 1c, relative Bayesian posterior model probabilities were.9996,.0004, < .0001 and <.0001 for null, small, medium, and large effects, respectively, indicating strong evidence in favor of H1c (BF01 = 2,826): that is, there is no effect of reporting errors on the quality ratings of the abstract. Given the high response among academics, we also provide results for academics only: Sample size: B = −0.26, SE = 0.11, t = −2.44, p = .015. Statistical significance: B = 0.06, SE = 0.11, t = 0.56, p = .575, reporting error: B = −0.03, SE = 0.11, t = −0.249, p = .803. The model with the three predictors explained 0.9% of the variance (95% CI: [.00,.02], F(3,683) = 2.08, p = .10). Relative Bayesian Posterior Model probabilities (H1c): .9984, .0016, < .0001 and <.0001 for a null, small, medium, and large effect respectively (BF01 = 629) All 800 participants were asked to name three aspects they considered relevant for their evaluation of manuscript quality. The 800 × 3 = 2,400 named aspects were classified in 17 categories. Sometimes all three aspects named by the same respondent were related to one category (e.g., study design), and sometimes one aspect was related to multiple categories (e.g., ‘Clear description of methodology’ is related to both writing and study design). Therefore, the number of categorized answers was not equal to 2,400. The minimum number of differently categorized answers by a participant was 0 (due to incorrect understanding of the question), and the maximum number of different categories was 4. This resulted in a total of 1,976 categorized answers. Table 5 shows the numbers of respondents naming an aspect related to each category, ranked by prevalence. Aspects related to sample size or power were mentioned by more than 25% of participants. Aspects related to statistical significance were named by only 14 respondents (1.75%) but 68 different respondents (8.5%) named the ‘results’ as an important aspect. Consequently, the maximum number of participants indicating that they were influenced by the strength of the results was 82 (10.25%). Statistical reporting error was mentioned by 13 participants (1.63%). The actual statistical reporting error was identified in only 3 cases (0.38%). Whereas sample size and statistical significance were likely to be commented upon in both the small/large sample size and the significant/non-significant result condition, mentioning (in)correct reporting was only likely for the 393 participants (49.13% of total sample) in the reporting error condition. 10.1371/journal.pone.0355938.t005Table 5Prevalence of Categories Indicated as Relevant for Assessing the Quality of the Abstract of Scientific Manuscripts and Students’ Theses (Text in Box 1), Provided by Students and Academics.PrevalenceMethodology/Research Design69.25% (n = 554)Writing (language, style, structure)35.13% (n = 281)Sample size/Statistical power25.13% (n = 201)Conclusions/interpretation of results21.50% (n = 180)Analysis (appropriate, missing tests, etc.)21.75% (n = 174)Relevance (novelty, impact, importance, etc.)17.00% (n = 136)Theory/Background14.00% (n = 112)Reporting of statistical results (e.g., effect size missing)13.75% (n = 110)Research question/hypothesis9.88% (n = 79)Results8.50% (n = 68)Sampling4.00% (n = 32)Statistical significance1.75% (n = 14)Statistical reporting error (correct or incorrect)1.63% (n = 13)Ethics1.38% (n = 11)Other (e.g., preregistration, open data)1.38% (n = 11) Prevalence of Categories Indicated as Relevant for Assessing the Quality of the Abstract of Scientific Manuscripts and Students’ Theses (Text in Box 1), Provided by Students and Academics. We ran a second regression analysis to investigate the interaction between the manipulated characteristics and manuscript type (thesis versus article, H2). Since students only received the thesis condition, only academics were included in this analysis. The regression model with only the four main predictors (sample size, significance, reporting error and manuscript type) explained 2.95% of the variance (F(4, 682)=5.19, p=.0004). The regression model adding the three two-way interaction terms between sample size, significance, and manuscript type (R2=2.99%, F(7, 679)=2.99, p=.004), shown on the left panel of Table 6, did not significantly improve the model, it explained only 0.04% more variance (F(3, 679)=0.10, p=.96). Because none of the interactions added to the explanation of quality rating, we cannot support hypothesis 2a or 2b; that is, sample size and statistical significance did not differentially affect quality rating of thesis versus articles for academics. To answer hypothesis 2c, Bayesian relative posterior model probabilities showed that an interaction effect of reporting error and manuscript type is not likely, confirming hypothesis 2c (Bayesian Posterior model probabilities: no effect: .9989, small effect: .0011, medium effect: < .0001, large effect: < .0001; BF01 = 869). 10.1371/journal.pone.0355938.t006Table 6Results of Regression Analyses Predicting Quality Ratings of Scientific Manuscript with Interactions of Manuscript Type and Career Phase. B (SE) t (p) 95%CIFit [95% CI] B (SE) t (p) 95%CIFit[95% CI]Intercept3.68 (.15)24.96 (<.001)[3.39, 3.97]4.04 (.15)26.94 (<.001)[3.75, 4.34]Sample−0.24 (.15)−1.60 (.111)[-0.54, 0.06]−0.27 (.15)−1.74 (.083)[-0.57, 0.04]Sig0.00 (.15)0.003 (.998)[-0.30, 0.30]0.11 (.15)0.72 (.474)[-0.19, 0.42]Error−0.03 (.15)−0.20 (.839)[-0.33, 0.27]−0.02 (.15)−0.14 (.890)[-0.33, 0.28]Manuscript type (MT)0.36 (.21)1.72 (.085)[-0.05, 0.77]Sample×MT−0.03 (.21)−0.13 (.894)[-0.45, 0.39]Sig × MT0.11 (.21)0.52 (.604)[-0.31, 0.53]Error×MT0.01(.21)0.04 (.965)[-0.41, 0.43]R2 = .0299[.00,.05]Career phase (CP)0.17 (.34)0.49 (.622)[-0.50, 0.84]Sample×CP−0.02 (.31)−0.05 (.958)[-0.63, 0.60]Sig × CP0.28 (.32)0.85 (.396)[-0.36, 0.91]Error× CP0.29 (.32)0.93 (.355)[-0.33, 0.91]R2 = .034[.00,.06]Note. Left side: Regression coefficients of sample size, statistical significance and reporting error and their interactions with manuscript type (MT, reference category is articles) on quality rating, academics only. Right side: regression coefficients of sample size, statistical significance and reporting error and their interactions with career phase (CP, reference category is academics) on quality rating, thesis only. Results of Regression Analyses Predicting Quality Ratings of Scientific Manuscript with Interactions of Manuscript Type and Career Phase. Note. Left side: Regression coefficients of sample size, statistical significance and reporting error and their interactions with manuscript type (MT, reference category is articles) on quality rating, academics only. Right side: regression coefficients of sample size, statistical significance and reporting error and their interactions with career phase (CP, reference category is academics) on quality rating, thesis only. Our third regression analysis investigated the interaction effects of the three manipulated design factors and career phase (student versus academic, H3; Sample×CP, Sig × CP, Error×CP). The regression model with only main effects explained 3.11% of the variance in quality ratings (F(4, 452)=3.63, p=.006), whereas the model containing both main effects and interactions explained 3.41% of variance in the quality ratings (F(7, 449)=2.27, p=.028), 0.3% more (F(3, 449)=0.47, p=.71). Interactions did not improve the prediction, see right the panel of Table 6. That is, neither sample size (H3a) nor statistical significance (H3b) showed an interaction with career phase in the quality ratings of theses. Bayesian posterior model probabilities supported the hypothesis that there was no interaction between reporting errors and career phase (H3c) since the null model received a probability of.9166 (small effect:.0834, medium effect: < .0001, large effect: < .0001; BF01 = 11). We ran a preregistered exploratory regression analysis on the full model predicting quality ratings of the manuscript, including all interaction terms up to four-way interactions. This full model explained 7.2% of the variance (F(23, 776)=2.61, p<.001), but of the predictors only sample size had an effect (B=0.69,p=.02). This full model was not beneficial compared to the model with only main effects (F(18, 776)=0.81, p=.68). A quantile regression as robustness check also showed no effects for H1 (neither for both academics and students, nor for academics only), but it did show an interaction effect between sample size and type of manuscript (H2a, p = .004). No effects for hypothesis 3 were found. Detailed results of the quantile regression can be found on OSF (https://osf.io/zpk67/). To summarize, we only found a weak effect of sample size on the quality rating of abstracts. Survey Study As the items in the survey did not correlate sufficiently (KMO-indices varying from 0.28 to 0.91) in all six conditions (Academics: Thesis quality, Thesis grading, Article quality, Article publishability. Students: Thesis quality, Thesis grading), we analyzed the data at the item-level for each condition separately to answer research questions 4 and 5 (see Table 7 and Fig 3). Table 7 shows the mean rating of each manuscript characteristic in each of the six conditions, as well as the average rating of each category (theory, design, conduct and analysis and presentation). Fig 3 also shows the importance rating for all 29 manuscript characteristics for all six conditions. The order of average importance of the four categories was the same for all six conditions. 10.1371/journal.pone.0355938.t007Table 7Mean Importance Ratings [1–7] of Manuscript Characteristics and Categories for Quality Assessment, in all Six Conditions (Students-Thesis, Quality & Grading; Academics-Thesis, Quality & Grading; Academics-Article, Quality & Publishability) (Standard Deviations Between Brackets).StudentsAcademicsTotalThesisThesisArticleQualityGradingQualityGradingQualityPubl.Describing the relevant theories (T1)5.85 (1.06)5.81 (1.08)5.87 (1.27)5.93 (1.16)5.59 (1.22)5.80 (1.13)5.80 (1.18)Deriving hypotheses from theory (T2)5.66 (1.32)5.59 (1.30)5.72 (1.28)5.65 (1.47)5.48 (1.24)5.52 (1.42)5.60 (1.35)Explicitly stating the hypotheses to the tested (T3)6.27 (1.20)6.22 (1.16)6.19 (1.09)6.15 (1.28)5.82 (1.29)5.93 (1.16)6.05 (1.21)Using a large number of literature references (T4)4.39 (1.41)4.48 (1.33)3.75 (1.45)3.92 (1.50)3.47 (1.44)3.44 (1.62)3.76 (1.52)Clearly describing the study objective (T5)6.36 (0.94)6.44 (0.88)6.43 (0.86)6.58 (0.83)6.37 (0.82)6.47 (0.75)6.46 (0.83) Theory 5.71 (1.39) 5.71 (1.34) 5.59 (1.53) 5.65 (1.56) 5.35 (1.57) 5.43 (1.63) 5.53 (1.55) Using an appropriate study design (D1)6.31 (0.84)6.44 (0.95)6.51 (0.81)6.51 (0.93)6.59 (0.72)6.64 (0.62)6.54 (0.80)Clearly describing the sample population (D2)5.83 (1.09)5.44 (1.25)5.93 (1.16)5.98 (1.14)5.85 (1.11)5.96 (1.09)5.89 (1.14)Reporting pre-study calculation of sample size (D3)4.47 (1.55)4.31 (1.53)4.42 (1.65)4.67 (1.56)4.10 (1.42)4.29 (1.58)4.38 (1.56)Having a large sample size (D4)4.90 (1.34)4.57 (1.45)4.28 (1.43)4.33 (1.46)4.58 (1.32)4.42 (1.40)4.45 (1.41)Using a complex design (D5)3.03 (1.27)2.80 (1.32)2.83 (1.63)2.85 (1.52)2.56 (1.45)2.65 (1.42)2.75 (1.48)Pre-registering (part of) the study (D6)4.78 (1.49)4.31 (1.54)3.70 (1.76)3.70 (1.56)3.52 (1.64)3.77 (1.62)3.80 (1.66) Design 4.89 (1.65) 4.65 (1.75) 4.61 (1.91) 4.67 (1.87) 4.53 (1.89) 4.62 (1.88) 4.63 (1.86) Achieving a high response rate (C1)4.25 (1.32)3.85 (1.45)3.91 (1.48)4.06 (1.41)4.13 (1.32)4.09 (1.32)4.05 (1.38)Achieving high statistical power (C2)4.41 (1.43)4.44 (1.41)4.44 (1.53)4.73 (1.40)4.95 (1.31)5.02 (1.40)4.74 (1.43)Applying complex statistical analyses (C3)3.61 (1.58)3.31 (1.33)2.93 (1.56)3.24 (1.66)2.91 (1.55)2.87 (1.51)3.06 (1.57)Providing open data (C4)4.42 (1.94)4.80 (1.76)4.10 (1.78)3.94 (1.81)4.07 (1.78)4.37 (1.67)4.19 (1.79)Providing open research materials (C5)4.68 (1.91)5.00 (1.78)4.40 (1.67)4.29 (1.83)4.34 (1.67)4.61 (1.63)4.47 (1.73)Providing open analysis (C6)4.47 (1.69)4.24 (1.87)3.94 (1.77)3.99 (1.73)4.04 (1.65)4.20 (1.59)4.09 (1.70) Conduct 4.31 (1.68) 4.27 (1.70) 3.95 (1.71) 4.04 (1.70) 4.07 (1.67) 4.20 (1.66) 4.10 (1.69) Adequately describing all used stat. procedures (A1)5.73 (1.42)5.89 (1.24)5.94 (1.23)6.13 (1.07)6.09 (1.03)6.03 (1.03)6.02 (1.13)Using the appropriate statistical analyses (A2)6.36 (1.11)6.56 (0.77)6.40 (0.98)6.42 (0.98)6.41 (0.87)6.52 (0.71)6.44 (0.90)Distinguishing conf. from expl. statistical analyses (A3)5.00 (1.69)5.33 (1.43)5.31 (1.31)5.38 (1.45)5.36 (1.49)5.71 (1.20)5.40 (1.41)Reporting assumption checks (A4)5.12 (1.27)4.87 (1.49)4.93 (1.40)4.97 (1.28)5.13 (1.25)4.90 (1.42)4.99 (1.34)Clearly presenting the statistical results (A5)6.20 (1.10)6.31 (0.99)6.35 (0.95)6.30 (1.04)6.31 (0.99)6.46 (0.73)6.34 (0.96)Providing CIs for the main stat. results (A6)5.10 (1.37)4.91 (1.26)4.92 (1.50)4.94 (1.48)4.99 (1.35)5.07 (1.44)4.98 (1.42)Providing effect sizes for the main stat. results (A7)5.29 (1.37)5.41 (1.32)5.51 (1.36)5.70 (1.36)5.60 (1.21)5.86 (1.17)5.62 (1.29)Providing statistical results without errors (A8)5.98 (1.32)5.70 (1.35)5.95 (1.29)6.19 (1.15)6.11 (1.33)6.18 (1.07)6.07 (1.24)Observing large effect sizes (A9)3.73 (1.83)3.41 (1.30)3.02 (1.69)3.22 (1.86)3.33 (1.72)3.53 (1.82)3.32 (1.76)Observing main effect in hypothesized direction (A10)3.97 (1.93)3.93 (1.90)3.25 (2.06)3.35 (2.13)3.47 (1.83)3.49 (2.06)3.47 (2.01)Reaching stat. sign. of results of main hypothesis A114.00 (2.17)3.09 (1.97)2.94 (1.98)3.03 (1.92)3.55 (1.92)3.46 (1.96)3.29 (1.99)Drawing correct conclusion from the stat. analysis (A12)6.44 (1.15)6.46 (1.06)6.63 (0.65)6.64 (0.89)6.63 (0.64)6.73 (0.50)6.63 (0.76) Analysis & presentation 5.24 (1.76) 5.16 (1.77) 5.09 (1.91) 5.19 (1.91) 5.25 (1.77) 5.33 (1.79) 5.21 (1.83) Total 5.06 (1.72) 4.96 (1.75) 4.84 (1.89) 4.92 (1.88) 4.87 (1.81) 4.97 (1.82) 4.92 (1.84) Mean Importance Ratings [1–7] of Manuscript Characteristics and Categories for Quality Assessment, in all Six Conditions (Students-Thesis, Quality & Grading; Academics-Thesis, Quality & Grading; Academics-Article, Quality & Publishability) (Standard Deviations Between Brackets). Mean Ratings of all Importance Ratings of Manuscript Characteristics for Quality Assessment for all six Conditions (see legend). On average, theory ranked highest on importance, closely followed by analysis and presentation (0.2 SD lower), with even lower average scores for design (−0.5 SD) and conduct (−0.9 SD). However, as characteristics within domains varied substantially, it is more meaningful to list those that were considered more important. These were (with average scores above 6): T3 (‘explicitly stating the hypotheses to be tested’), T5 (‘clearly describing the study objective’), D1 (‘using an appropriate study design’), A1 (‘adequately describing all used statistical procedures’), A2 (‘using the appropriate statistical analyses’), A5 (‘clearly presenting the statistical results’), A8 (‘reporting the statistical results without errors’), and A12 (‘drawing the correct conclusion from the statistical analysis’). Considered less important (average scores below 4) were: T4 (‘using a large number of literature references’), D5 (‘using a complex design’), C3 (‘applying complex statistical analyses’), A9 (‘large effect sizes’), A10 (‘effect in hypothesized direction’), and A11 (‘statistical significance’). We also listed characteristics related to the characteristics we manipulated in the vignette part of this study. For sample size, these were characteristics D4 (large sample size) and C2 (high statistical power). In all six conditions, these characteristics were rated as moderately important for quality assessment. The characteristics related to statistical significance (A9-A11; large effect sizes, effect in hypothesized direction, statistical significance) were rated as having little importance, whereas the characteristic related to reporting error (A8, ‘reporting the statistical results without errors’) was of high importance in all conditions. Table 8 shows the correlations between the mean ratings for the conditions we compared in research questions 6–13 (see Fig 1), to establish the pair-wise association of all 29 characteristics between pairs of conditions. Correlations were high (r > .9), indicating strong association between mean importance ratings, which is also confirmed by visual inspection of Fig 3. Participants in all conditions seemed to agree with respect to their importance ratings of the different characteristics, regardless of manuscript type (thesis or submitted article), type of evaluation (quality or grading/publishability), and regardless of career phase (students or academics). Correlations between on the one hand thesis quality and thesis grading and on the other hand between article quality and article publishability were all very high (>.93, most > .99) and hence not distinguishable (RQ10). Within-category correlations were generally less strong for the research conduct category. 10.1371/journal.pone.0355938.t008Table 8Correlations Between Mean Importance Ratings Within Categories (Theory, Design, Conduct, Analyses & Presentation) and Across Categories (Total).RQ 6RQ 7RQ 8RQ 9RQ 11RQ 12RQ 13AcademicsStudentsThesis QualityThesis GradingThesis Quality- GradingArticle Quality- Publish.QualityThesis-ArticleThesis Grading -Article Publish.Thesis Quality- GradingStudents- AcademicsStudents- AcademicsTheory.995***.998***.995***.999***.996***.992***.989**Design.998***.996***.990***.992***.986***.942**.956**Conduct.934*.982***.953**.964**.905.937*.663Analysis & presentation.998***.992***.996***.994***.971***.981***.984*** Total .996***.994***.982***.982***.971***.972***.962***Note. * indicates p < .01, ** indicates p < .005, *** indicates p < .001 Correlations Between Mean Importance Ratings Within Categories (Theory, Design, Conduct, Analyses & Presentation) and Across Categories (Total). Note. * indicates p < .01, ** indicates p < .005, *** indicates p < .001 Although correlations were very high, average importance ratings of manuscript characteristics sometimes differed across conditions, as can be seen in Fig 3 as well as Table 7. Focusing on differences of at least 0.5 standard deviation ((SD1+SD22), using a large number of literature references (T4) was considered more important by students for the quality and grading of theses than by academics for the quality and publishability of articles. Interestingly, preregistration (D6) was rated by students as more important for the quality of a thesis than by academics for anything (all four conditions), and as more important by students for the grading of theses than by academics for the quality of an article. Discussion Peer review is an important gatekeeper of the quality of published scientific literature. However, little is known about the manuscript characteristics reviewers consider most important when evaluating manuscript quality, whether academics differ in their peer review and their evaluation of master’s theses, and if students differ from academics in their evaluation of scientific manuscripts. This study therefore aimed at getting more insight in the manuscript characteristics that are most relevant to students and academics when they assess the quality of scientific manuscripts. Our study showed that sample size had a small positive effect on quality assessment. In the survey study, respondents considered sample size only of moderate importance. Statistical significance did not influence quality assessment in our vignette study. Only a small number of respondents considered it a relevant aspect. In the survey study, characteristics related to the magnitude, direction, and statistical significance of the results were rated of low importance. Especially perceived lack of importance of statistical significance in quality assessment is surprising, since published research often report statistically significant findings in a context of low statistical power. The overabundance of statistically significant results in the literature suggests that statistical significance plays a larger role in manuscript submission but a smaller role in quality assessment. This is consistent with Augusteijn et al. [47], who found that statistical significance of the main result did not predict thesis grade. We, however, should be careful generalizing the conclusions based on our results of our vignette study. An abstract that only lists the hypotheses and results might not be a sufficient proxy for an entire manuscript. Reviewers’ responses might be different to an article that builds towards a specific hypothesis and provides a nuanced discussion of the results. However, we anticipated that asking subjects to read complete research articles would result in a sample too small to obtain reliable results and thus we chose to present them only with abstracts. The lack of importance of statistical significance found in the survey may also be due to socially desirable answering since researchers might believe that statistical significance should not matter. And finally, at a more general level, other manipulations of statistical significance, sample size, and statistical reporting error may also lead to other results than we found in our vignette using an abstract of a scientific manuscript. In our vignette study, the reporting error did not influence quality rating. According to the survey study, however, error-free reporting was of high importance. Perhaps participants did not spot the reporting error in the abstract, so that it did not affect the quality ratings. Schroter et al. (2008) found that reviewers have a hard time detecting errors in studies, explaining the high prevalence of reporting errors in the literature [54]. Tools such as statcheck, which automatically extract statistics from articles and recomputes p-values based on test statistics [55] may be valuable. Statcheck could assist the reviewers and editors in detecting such errors. Another possible solution could be to include statistical experts earlier in the research process [56,57]. Ideally already assisting when planning the research, but also in the review process. No interactions were found between the three manipulated characteristics and the text type (thesis or submitted manuscript, hypothesis 2) or career phase (student or academic, hypothesis 3). We found that overall quality rating of manuscripts varied considerably (Table 4), confirming lack of agreement in peer reviews found in other research projects [38–40]. Participants varied in the aspects they considered influential to their quality rating and even contradicted one another. For example, some respondents considered the abstract excellently written while others considered the same writing very bad. Likewise, some considered the choice of manipulation was poorly explained but others considered the same manipulation to be very good. We did not study the possible explanation of these differences but conclude that quality assessment varies between reviewers, so that the results support the popular conjecture that the editor’s choice of reviewers can affect acceptance or rejection of a manuscript for publication. Our survey revealed large agreement of the importance ratings between the respondents in the different conditions. Importance ratings of the different characteristics were similar for theses and submitted articles, students and academics, and different types of evaluation. This suggests that academics uphold similar quality criteria for master’s theses as for manuscripts submitted for publication but the bar is set lower for master’s theses. Students successfully gauged the characteristics important to their supervisors, suggesting that academics successfully passed on their own quality criteria to students. Krishna and Peter [46] showed that students’ attitude and behavior towards QRPs and RRPs is determined by the attitude they perceive from their teachers. Others found that type of mentoring influenced the probability of engaging in QRPs and RRPs [58–60]. Assessments of quality and of publishability of an article were highly similar with respect to importance of characteristics. This is surprising, since it is a common conception that there are more interests at play for publication than research quality alone [(e.g., 33, 61)]. Characteristics related to theory were considered most important in all conditions, whereas characteristics related to research conduct were rated least important. Most important individual characteristics were ‘drawing the correct conclusions’ (A12), ‘appropriate study design’ (D1), ‘describing study objectives’ (T5), and ‘using appropriate statistical analysis’ (A2). Except for describing the study objective, these characteristics are hard to judge objectively: when are conclusions warranted, when are the study design and statistical analysis appropriate? Many-labs and multi-analyst studies have shown that the same dataset and research question give rise to various ways of data analysis, rendering assessment whether data were analyzed in ‘the appropriate way’ subjective [62]. The least important characteristics were ‘complex design’ (D5), ‘complex analysis’ (C3), ‘large effect sizes’ (A9), ‘effect in hypothesized direction’ (A10), and ‘statistical significance’ (A11). The fact that appropriateness of the design and the analysis were rated most important while complexity was rated least important is a positive result. Likewise, characteristics related to the strength of the results were rated least relevant. However, the literature shows an overabundance of statistically significant results, and researchers’ use of QRPs suggests that what people believe is important is not necessarily what is reported. Atkinson et al. [45] found similar results in their experimental study. There could be a difference between the characteristics researchers believe influences them and the characteristics that really do [63], and social desirability of answering may play a role too. Limitations and future research Because we were unable to sample enough students, thus resulting in a lower statistical power, all analyses, hypotheses, and conclusions including a comparison with students should be considered exploratory. We aimed at achieving a power of .80 but only achieved a power of .18 for a small effect (f = .10) for those tests that only include students. Our results strongly suggest that students evaluate the quality of manuscript characteristics like researchers, but larger sample sizes are recommended to confirm these exploratory results. However, we also should be careful drawing general conclusions on the similarity of the evaluation of researchers and students, as researchers and students were recruited with different methods that may have resulted in sampling bias. The academics sample enabled a power of .74 to detect a small effect, whereas the analyses including all 800 respondents had a power of .81. The samples of academics and students may be biased, possibly due to students and academics participating who are interested above average in meta-science and concerned with good research practices. The abstract we used in the vignette may not be a valid proxy for a scientific article. Readers often use an abstract alone to assess the value of an article [27], and therefore future research might focus on the issue of whether quality assessment of an article based on an abstract alone is a good predictor of the quality of the whole manuscript. Our survey study might have lower ecological validity than studies with an observational or experimental setup. Unlike participants in [34] and [45], our participants were aware that they were participating in a study. The study characteristics academics find important to the manuscript’s quality are not necessarily the characteristics that influence the assessment the most when peer reviewing a manuscript. For instance, complex design and complex analysis were considered least relevant for manuscript quality in our survey, but Armstrong [42] suggested that reviewers are influenced by these false cues. In our vignette study, we manipulated only three aspects and presented participants with just an abstract. An experimental setup with a manipulation of all 29 aspects we investigated in our survey does not seem feasible. Future research could investigate the impact of some of the characteristics that are considered either irrelevant (e.g., complex design and complex analysis) or highly relevant (e.g., (in)correct conclusions, (in)appropriate study design, describing study objectives, and using (in)appropriate statistical analysis). The aspects that were considered highly relevant are also aspects about which reviewers often disagree. For example, the assessment of the appropriateness of the study design and the appropriateness of the analysis seem to be subject to judgment. To gain more insight in the characteristics that increase the publishability of a manuscript, one might investigate whether the literature is homogeneous on aspects considered highly relevant (e.g., clearly describing the study objective) and heterogeneous on aspects with little relevance (e.g., the number of references). This could provide an indication that the highly relevant characteristics are a minimum requirement, whereas characteristics with little relevance are indeed arbitrary to the publishability of an article. Finally, it would be interesting to know how our results relate to results from other fields than psychology, and how the importance of characteristics might change over time. For example, the popularity of open science and sharing materials is increasing and might in part explain the relatively large variance in the importance rating of this characteristic amongst academics. 18 Mar 2025 Dear Dr. van Assen, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. In your review, please provide detailed answers, particularly regarding the statistical analyses and the application of scale, domains, and scores, as discussed by both reviewers. Among Reviewer #2’s concerns is the use of frequentist analysis for most of the study without reasonable explanation. Please explain the reasoning behind choosing this statistical method for your hypotheses. Both reviewers have also expressed skepticism about sample validation, recruitment, and data interpretation. Reviewer #1 is also concerned about how readers should interpret your scores and criteria to assess the quality of abstracts, manuscripts, and theses, as they do not necessarily reflect peer review practices. Please submit your revised manuscript by May 02 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Sonia Vasconcelos, PhD Academic Editor PLOS ONE Journal Requirements: 1. When submitting your revision, we need you to address these additional requirements.-->--> -->-->Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at -->-->https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and -->-->https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf-->--> -->-->2. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. -->--> -->-->When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.-->--> -->-->3. Thank you for stating the following financial disclosure: -->-->Research Talent Grant of the Dutch Science Foundation (NWO).-->--> -->-->Please state what role the funders took in the study. If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." -->-->If this statement is not correct you must amend it as needed. -->-->Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.-->--> -->-->4. Please note that your Data Availability Statement is currently missing the repository name and/or the DOI/accession number of each dataset. If your manuscript is accepted for publication, you will be asked to provide these details on a very short timeline. We therefore suggest that you provide this information now, though we will not hold up the peer review process if you are unable.-->--> -->-->5. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered into the online submission form will not be published alongside your manuscript.-->--> -->-->6. We note that you have referenced “Cooper H, DeNeve K, Charlton K.” which has currently not yet been accepted for publication. Please remove this from your References and amend this to state in the body of your manuscript: (ie “Cooper H, DeNeve K, Charlton K.” [Submitted]”) as detailed online in our guide for authors-->-->http://journals.plos.org/plosone/s/submission-guidelines#loc-reference-style-->--> -->-->7. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.-->?> [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? Reviewer #1: No Reviewer #2: Partly ********** 2. Has the statistical analysis been performed appropriately and rigorously? -->?> Reviewer #1: No Reviewer #2: No ********** 3. Have the authors made all data underlying the findings in their manuscript fully available??> The PLOS Data policy Reviewer #1: Yes Reviewer #2: Yes ********** 4. Is the manuscript presented in an intelligible fashion and written in standard English??> Reviewer #1: Yes Reviewer #2: Yes ********** Reviewer #1: Thank you for the opportunity to review this manuscript. The following are my recommendations for its improvement: Title 1. As this is not about evaluation of actual manuscripts, I would suggest changing the title to: Exploring students and researchers rating of the quality of manuscripts Abstract 1. Please include information on the quality scores given to vignettes, and quantify the influence of the sample size you mention. Additionally, specify if these were actual existing abstracts/manuscripts, or created for this experiment 2. These results suggest that researchers’ assessments of manuscript quality are not 24 responsible for publication bias. – I disagree this study was not designed to assess publication bias. 3. “are already adopted” – I find this conclusion wrong, this might indicate that science educations implies certain standardized quality considerations. I do not find adopted as a proper word in this context. We know from rant review, that when we specify domains and provide concrete ratings, researcher can apply those, same as teachers in grading students or their coursework. Why would researchers need to adopt anything, when they are not ask to rate manuscripts in this way in everyday research publishing or review. Introduction 1. I believe authors have to discuss the influence of registered reports on the reporting of non-significant results, and current data on this, similarly they can comment on the results of clinical trials that have to be registered by law – i.e. researchers may chose to pursue studies for which their pilot analysis already indicate potential positive effect. Peer Review 1. There are approximately 33,100 peer-reviewed – Please provide citation for this, current estimates are around 70,000 using Open Alex data. Similarly we have more than 6 million articles published per year, using OpenAlex or Dimensions 2. As the authors stated “we focus on the question which manuscript characteristics are most important for manuscript quality rating that might differ between reviewers and even between manuscripts.” – In regards to this I would greatly revise the current introduction and peer review section, and provide literature review on published papers that aimed to assess manuscript or peer review quality, and report on methods used to do so, and domains assessed in previous research. 3. “also tested what influenced their quality assessment” – was this testing part of the original study when the original question was asked? If not, this should be clearly stated. Participants 1. Please report what was the original plan for not meeting the sample size. You stated you needed at least 787 participants, and you obtained 687 – why did you not conduct another or several more rounds of invites? 2. “ even when their main field of research was not within a research field in psychology” – Please spicy if this was originally planed like this, or were you aiming to collect responses only from individuals with primary field of psychology. 3. Due to later statement “in which psychology research field they primarily worked”, I would appreciate a sub-analysis of results, with those not primarily working in psychology being excluded, and compared to results of those that do. 4. Table 1, I would suggest that the reporting is changesdto N (%) with 0 decimal points, or on the reverse order, the use of n=, seems unnecessary. 5. Table 3 – I dislike the phrasing - Using a large number of literature references., I would find this to have a QRP associated with it, Using relevant lit. references seems more important than the number. Similarly phrasing of using complex design and using complex analyses, seem out of place in the list. The researchers should be aware of the pyramid of evidence, which of those do the drafters consider to the complex? Why Is using complex analysis in C, when using appropriate analysis is in A – this seems to be bad design of the scale and domains. What reliability testing has been done to confirm that these items belong under the same domains? It seems evident from the way respondents answered the question on what influences the ratings, that they might not adhere to the domains your teams have devised. In a registered report or protocol with statistical analysis plan predefined, is using the appropriate statistics part of design or only of analysis? Why is providing code or data part of conduct and not of presentation? Why is describing the sample part of design and not presentation? You had a planned sample, and you did not reach it. The sentence “Clearly describing the sample population” – can only occur after analysis of the results, so it can not be part of design as Table 3 would imply. In short, please indicate in the survey itself, were respondents presented the sentences with the domains or only sentences? I was not surprised to read later that the KMO scores were low, as these domains to me do not meet face validity. 6. Since hypotheses 1c, 2c and 3c indicated no effect, we evaluated these hypotheses using Bayesian statistics. – Please indicate if this analysis was preregistered? If not, this should be moved to appendix. 7. I would suggest putting content coding as a subtitle in the analysis section Results 1. Please indicate in the stat. section if the quality rating data was normally distributed to report M and SD, instead of Md and IQR? 2. Reporting of aspects – would it be possible for you to showcase the first choice of the most important aspect of participants, then the second, then the third? I am not sure that the coding you applied matches the domain coding of researchers themselves. You separated Reporting of statistical results, statistical significance, and statistical reporting of error, from results reporting. And yet in Table 3, these items fall under domains of conduct or analysis and presentation. I would have expected your own top domains to be identical. You state “other (e.g. preregistration, open data)”, but in Table 3 this is part of conduct, which I disagree with conceptually, to me this is part of presentation, but at least you should apply your own consistent coding and justification behind it. 3. Results style – As you had 13 research questions, please rewrite the results in the order of those questions, with appropriate subtitles and report them in that order. I would highly recommend, that you start with, results for our 13 predefined research questions are provided in Table X or Box X, which would list each question, the answer to it, as well as strength of evidence rating for it, with indication or planned vs additional analysis to explore/confirm the RQ. 4. “As the items in the survey did not correlate sufficiently” – Please start this section first with a little explanation. To test X, we conducted KMO analysis which indicated that the prespecified domains were no…. As preplaned, we then analysed data at the item level. “the average rating of each category “ – why is this shown, when you have evidence that they do not belong to the same category? 5. “Theory was rated to be the most important” – I am not sure that this is the proper way to interpret these results. As you decided on item level presentation, list the items with most. One of the design items had a very high score too, and readers should know which of the individual items statistically differ in scores from which items? In regards to the vignette, where answers indicated that design is by far the most important aspect (when asked to list 3 aspects), we as readers should know if this second type of rating had stat. sign difference between which individual level items. 6. Table 7 – average domain level scores should be deleted, due to KMO scoring indicating these are not the same domain. Those that had appropriate scores, can be considered for average score, if you feel that appropriate. Discussion 1. I will refrain from detailed comments on discussion until the above are clarified. I would however advise authors to make it very clear that researchers are not normally asked to rate the quality of abstract/manuscripts/thesis using the scale presented in this domain. And the peer review of the manuscripts does not work in regards of improving the initial given score, to then a higher score, as many aspects of the design can rarely be changed. So I would suggest emphasizing in the introduction and discussion, that in this exercise of trying to score manuscripts, what was found. Also, we have no test-rest scores for this, and we are not sure, that the same researchers would give the same score or not change their criteria when asked to do these kind of tasks regularly. In hopes the above can help improve the readability and presentation of the results. Reviewer #2: The manuscript describes an interesting empirical attempt at studying some aspects of how reviewers evaluate articles or theses. Unfortunately, there are some important limitations that are not adequately addressed by the authors. I also have some issues concerning the statistical analyses for some of the hypotheses under study. Major concerns: 1. As far as I could understand, the entire vignette study was based on a single abstract, with three elements (sample size, statistical result and reporting error) that varied between two different categories. This poses an important restriction on any generalizations that can be drawn from the study, as one cannot generalize from this specific case how much judgments of quality by researchers or students are affected by these three factors in general. For instance, this is only one example of a reporting error (which may not be particular evident for most people): other reporting errors may be more easily noticed and have a much larger impact on quality judgments. Similarly, studies with different designs or topics may have their quality assessments more or less affected by sample size or statistical results. Generalizing from the experimental study to general statements concerning what researchers take into account when evaluating manuscript would likely require multiple examples, which would ideally be entered into a statistical model as random variables (for more on this issue, see Yarkoni, 2020 - https://doi.org/10.1017/s0140525x20001685). This seems to me the most important limitation on what can be concluded from the study – however, it does not seem to be mentioned at all in the limitations section. The impossibility of generalizing from this single abstract to manuscripts/theses in general should be clearly stated both in the introduction and discussion, and taken into account when presenting the results. This is particularly important for the statistical reporting error variable, the detection of which seems to require that participants would be able to spontaneously notice that a p value does not correspond to a given t value (a correspondence which most researchers are not used to doing without the use of software or tables). I would thus strongly avoid making any kind of generalization about how much researchers perceive or value reporting errors based on this example alone. 2. The recruitment of researchers and students is done through very different methods: thus, any statistical comparisons may be reflecting differences in the populations induced by the recruitment methods, rather than the different career stages. Particularly, the fact that students were recruited through a method that was much more dependent on the personal contacts of the researchers raises concern that this may have selected a population that was more familiar with discussions on open science/reproducibility than the general population of researchers recruited through the Web of Science. This should also be explicitly mentioned in the discussion section and in any comparisons drawn between researchers and students throughout the manuscript. 3. Research questions 6 to 13 seem not to have been addressed by the statistical analysis at all. Although the formulation of these questions is rather haphazard in the manuscript (which makes it difficult to understand some of them), the authors state that they will “study differences in the relevance of the different characteristics” when assessing quality vs. publishability, theses vs. articles, etc. However, the only statistical analyses presented for these findings are correlations between the answers in the different conditions (academics vs. students, articles vs. theses, quality vs. grading/publishability). These do not test for the presence of differences between these conditions at all, as the presented p values for correlations test the (highly unlikely) null hypothesis that there would be no correlation at all between answers in both conditions, and don’t answer the intended question of whether there are differences in the assessment of different features between the conditions at the population level. To answer these questions, a direct statistical comparison between the rating of different items in different conditions would be needed, in order to test the null hypothesis that there is no association between conditions and survey answers. If the authors feel that the exploratory nature of the questions and the large number of items/conditions do not warrant formal significance testing, this is fine. However, it would still be useful to present p values or confidence intervals for differences. Moreover (and perhaps more importantly), statements such as “This suggests that academics uphold the same quality criteria for master’s theses as for manuscripts submitted for publication” are clearly not warranted based on a correlation analysis alone. 4. I find it quite strange to use traditional frequentist analyses for most of the article, but a Bayesian analysis for hypothesis 1c/2c/3c. If I understood correctly, this was done because the authors’ hypothesis was that there would be no effect, and thus they wanted to provide evidence in favor of the null hypothesis. Doing this through Bayesian analyses, however, eliminates the commensurability between the analyses used for different hypotheses, as the conclusions that can be drawn from Bayesian vs. frequentist approaches are fundamentally different. The paper would thus be more coherent if either Bayesian or frequentist analyses (or both) were used for all hypotheses. Arguing in favor of the lack of a large effect shouldn’t be an obstacle to use frequentist analysis: if the authors want to evaluate evidence against an effect, they can use traditional equivalence testing for this to remain within the framework used in the rest of the article. Conversely, there is no problem in using Bayesian analysis to test directional hypotheses such as 1a/b, 2a/b and 3a/b. Minor concerns: General: 1. Figure legends seem to be missing from the manuscript file altogether (or at least I could not find them). 2. For some reason, line numbers disappear after page 29. Introduction: 1. The introduction is very long, with a lot of general sections about peer review that frequently touches on issues that are general knowledge for most scientists and are sometimes not directly related to the research questions. I personally found this literature review quite interesting and thorough, but also somewhat superfluous. This is a matter of style and the authors do not need to change it, but I would consider shortening the introduction to make the paper more readable. Study hypotheses: 1. The exploratory research questions (RQ4-RQ13) are formulated haphazardly and are hard to understand clearly either from the text or from Figure 1, which is hardly self-explanatory. Please consider formulating them one by one, as was done for the first research questions. Also note that RQ4 and 5 are not mentioned in the figure. 2. The correspondence between statistical analyses and hypotheses is also not clear. It seems to me that RQ2 and RQ3 are being tested through the interaction terms in the regression models, for example but it would be useful to state this clearly in the hypothesis or methods section. Methods: 1. I found it confusing that PhD students seem to be considered as “researchers”, while the “student” category is reserve for bachelor and master students. I have no problem with this somewhat arbitrary category, but it should be made clear in the methods (until getting to Table 1, I was assuming that PhD candidates would be considered students). 2. Sample size and statistical results can certainly interact in driving quality assessments (i.e. “could not explain the sex differences” in the vignette because of a p = 0.6 is a more appropriate conclusion when sample size/statistical power is high, but may be too strong a conclusion when it is low). Were interactions between the three manipulations tested in the first regression model? I assume they are present in the “full model”, but I could find no direct results concerning them, and I’d be curious about them. 3. Was the vignette presented before or after the survey? I got the impression that it came before, but given that one can influence the other it is important to make this clear. 4. The “conduct” category in the survey is a bit of a hodgepodge. I don’t really see what “providing open data, materials, etc.” has to do with “achieving a high response rate” (which is independent of the researchers) or “achieving high statistical power” (which depends on items mentioned in the design section”). There is no way to go back on this now, but it may be worth mentioning in the limitations section. Results: 1. For the vignette study, I don’t think it is very useful to look at the effect sizes for each factor in each group separately (Students, Academics/Thesis, Academics/Article), as is done in Table 4, without looking first at the main effect of these factors in the whole population (for which there is a much larger sample size and statistical power) – which only happens afterwards in the manuscript. Moreover, presenting these factors separately gives the impression of variation that may be just noise from these relatively small sample sizes (and nothing in the table allows one to evaluate this). Please consider presenting the regression results first, and then presenting the effect sizes separately for each group (with p values for the interaction between manipulations and conditions, which currently appear in Table 6 but not in Table 4) to allow the reader to judge whether the variation is above what would be expected by chance. 2. Please replace asterisks by exact p values whenever possible (e.g. Table 4). 3. Table 6 is quite hard to understand. I could eventually make out the meaning of each term, but terms such as “constant”, “sample”, “sig”, “error”,, “Sample*MT” etc. should be defined in the legend (or expressed in reader-friendly terms in the table itself). 4. Similarly, Table 7 is only comprehensible to the reader if he refers back to Table 3 (which specifies the meaning of T1, etc.). I think it would become much more reader friendly if the description of each item was included. If space is a problem, it’s probably worth including only the totals (as the differences between conditions are minor), leaving the six separate conditions only in Fig. 3, which provides a more intuitive way to look a the comparisons. 5. In Figure 3, the X axis should include the category to which the data refers to. Specially for the analysis and presentation category, it is quite annoying to have to count in order to discover which category each point refers to. Once more, brief descriptors would be more useful than T1, T2, etc., but space may be an issue in this case. 6. Table 8 could probably be omitted or moved to the supplementary material, as the close correlation across conditions can be visualized in Figure 3, and knowing the exact correlation value for each of them does not add much, and doesn’t really help to answer the research questions 6-13 (see major concern # 3) Discussion: 1. The last paragraph of the article is a confusing mix of different subjects and could be replaced by a more focused conclusions section. Point issues: 1. Line 104 – there is a loose parenthesis. 2. Line 156 – “For example, the sample size of what students believe to be a large enough” – this sentence is odd (i.e. a large enough what?). Please revise. 3. Line 160 – When quoting numbers such as “over 90% of psychology researchers admitted having engaged in QRPs”, it’s worth stating more precisely what is considered under the “QRPs” umbrella. 4. Line 175: “We did not only investigate the believes”. “Beliefs”, not believes. 5. Line 180: What the authors meant by “Statistical reporting error” was not clear to me at this point, perhaps it’s worth defining it better (e.g. “the presence of discrepancies in statistical values” or something like that). 6. Line 206: “differs between students and academics”, not “from between”. 7. Box 1: The red color used to highlight the last manipulation makes it hard to read the text, consider changing to a lighter hue. Also, I’d argue that this is technically a figure and not a text box. 8. Line 458. “All 800 participants were asked to name three aspects they considered relevant for their evaluation of the manuscript quality.” – consider specifying “to the quality of the abstract presented in the vignette” to make it clear that the open responses concern the vignette and not manuscript quality in general. 9. Discussion: “We expect reviewers consider sample size of higher importance than the participants in this study did, given that small sample size may be considered a red flag in quality assessment of an article, particularly in combination with a (barely) statistically significant outcome, which more likely represents a false positive.” I could not understand this statement. Do the expectations mentioned concern reviewers in general? 10. Discussion: “In the current study, reviewers did not seem to be the main source of publication bias.” This is a weird statement: there is no publication decision being made un in the study, so it seems strange to talk about “publication bias”. ********** what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy Reviewer #1: Yes: Mario Malicki Reviewer #2: Yes: Olavo B. Amaral ********** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. 11 Sep 2025 See our response letter. Submitted filename: response letter PLOS ONE.docx 7 Mar 2026 Dear Dr. van Assen, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Apr 21 2026 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. A letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Sonia Vasconcelos, PhD Academic Editor PLOS One Journal Requirements: If the reviewer comments include a recommendation to cite specific previously published works, please review and evaluate these publications to determine whether they are relevant and should be cited. There is no requirement to cite these works unless the editor has indicated otherwise. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author Reviewer #1: (No Response) Reviewer #2: (No Response) Reviewer #3: All comments have been addressed ********** 2. Is the manuscript technically sound, and do the data support the conclusions??> Reviewer #1: No Reviewer #2: Partly Reviewer #3: Yes ********** 3. Has the statistical analysis been performed appropriately and rigorously? -->?> Reviewer #1: No Reviewer #2: Yes Reviewer #3: Yes ********** 4. Have the authors made all data underlying the findings in their manuscript fully available??> The PLOS Data policy Reviewer #1: Yes Reviewer #2: Yes Reviewer #3: Yes ********** 5. Is the manuscript presented in an intelligible fashion and written in standard English??> Reviewer #1: No Reviewer #2: Yes Reviewer #3: Yes ********** Reviewer #1: Dear authors, thank you for the revision, but I find that you did not address my comments in the way I would want them, and so I advised the editor to get an extra statistical reviewer to weigh in on differences between my requests and your approaches. Reviewer #2: The manuscript has addressed many of the concerns raised previously by both reviewers, especially concerning issues in the text and tables. That said, I think some of the major concerns mentioned in the previous review remain unaddressed. I will focus this review on these main points. 1. I don’t think my main concern (major concern #1) has been addressed by the authors. They did include remarks that one should be careful of generalizing the conclusions based on evaluation of an abstract to evaluation of whole manuscripts or theses, which is appropriate. But they still fail to comment on the (to me, much more important) limitation that the conclusions relating to the vignette study are based on evaluation of a single, specific abstract that was presented to all participants, with a single, specific manipulation of each factor. Thus, the analysis does not allow one to generalize the conclusions even to other abstracts or other manipulations. It may well be the case that, for different statistical errors, different p values, different results, or different abstract topics, the results found would be very different. This is the point I was trying to get to in my previous review: I still believe it is a very important limitation, and it is still not mentioned anywhere in the discussion. 2. I still think that a correlation analysis between the responses of different groups of researchers (e.g. students vs. academics, those evaluating “theses” vs. those evaluating “articles” ) maps poorly to research questions 6 to 13, which purportedly aim to study differences between groups or conditions. I am sensitive to the authors’ argument that studying the overall similarity of answers via correlations is more economical for presenting the results, as evaluating pairwise differences between answers for different topics would generate a lot of numbers to look at. But if the authors are only measuring correlations, they should be clear that the results mean that answers across groups and conditions are highly correlated, not that they are “the same”, as is used in the discussion (e.g. page 38, line 9, as well as multiple times in the responses to reviewers). There is still space for single items to vary considerably within a correlation of 0.9, and some differences in Table 7 appear to be quite unlikely to have arisen by chance given the sample size, even though no statistics are provided (e.g. preregistration has a quality rating of 4.89 for students vs. 3.70 for academics). The authors may refrain from doing inferential statistics to evaluate these differences if they wish, but in this case they should (a) note that differences in individual items do occur between populations/conditions, perhaps citing the most obvious ones and (b) acknowledge in the discussion that the analysis is not really looking into differences between importance ratings as stated in the discussion, but rather assessing the overall similarity between the relative importance of different items. Alternatively, they may also actually evaluate differences in individual items using inferential statistics. 3. I still think that the presentation of questions 4 to 13 in the introduction within a single text block is confusing. I’d strongly suggest that the authors list these individually, as they did for questions 1-3. Figure 1 is not immediately intuitive to the reader to fill that gap, and the brief legend does not help that much). 4. If interactions between sample size and statistical significance in driving results were not tested (i.e. previous Methods point #2), this should be mentioned as a limitation in the discussion as well. I would certainly have an expectation that statistical significance may have different effects on quality judgments depending on sample size, as the appropriate interpretation for a given p value in terms of concluding about presence/absence of an effect should depend on the statistical power of the study. Minor points: Table 6 – Career, not carreer Reviewer #3: Dear authors, I had some concerns. First, whether the study evaluated internal consistency using Cronbach's alpha or McDonald's omega . Did you create the instrument or, in this case, did you use it from a specific author and the information from the instrument? If the instrument was created by the authors, was there any type of validation by expert judgment, followed by a pilot study and, therefore, internal consistency and instrument validity using an AFC and an AFE? ********** what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy Reviewer #1: Yes: Mario Malicki Reviewer #2: Yes: Olavo B. Amaral Reviewer #3: No ********** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] To ensure your figures meet our technical requirements, please review our figure guidelines: https://journals.plos.org/plosone/s/figures You may also use PLOS’s free figure tool, NAAS, to help you prepare publication quality figures: https://journals.plos.org/plosone/s/figures#loc-tools-for-figure-preparation. NAAS will assess whether your figures meet our technical requirements by comparing each figure against our figure specifications. 22 Apr 2026 See the included document Submitted filename: response letter PLOS ONE revision_HA.docx 10 Jul 2026 Dear Dr. van Assen, Please submit your revised manuscript by Aug 24 2026 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. A letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. As the corresponding author, your ORCID iD is verified in the submission system and will appear in the published article. PLOS supports the use of ORCID, and we encourage all coauthors to register for an ORCID iD and use it as well. Please encourage your coauthors to verify their ORCID iD within the submission system before final acceptance, as unverified ORCID iDs will not appear in the published article. Only the individual author can complete the verification step; PLOS staff cannot verify ORCID iDs on behalf of authors. We look forward to receiving your revised manuscript. Kind regards, Sonia Vasconcelos, PhD Academic Editor PLOS One Journal Requirements: 1. If the reviewer comments include a recommendation to cite specific previously published works, please review and evaluate these publications to determine whether they are relevant and should be cited. There is no requirement to cite these works unless the editor has indicated otherwise. 2. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author Reviewer #2: (No Response) Reviewer #3: All comments have been addressed ********** 2. Is the manuscript technically sound, and do the data support the conclusions??> Reviewer #2: Partly Reviewer #3: Yes ********** 3. Has the statistical analysis been performed appropriately and rigorously? -->?> Reviewer #2: Yes Reviewer #3: Yes ********** 4. Have the authors made all data underlying the findings in their manuscript fully available??> The PLOS Data policy Reviewer #2: Yes Reviewer #3: Yes ********** 5. Is the manuscript presented in an intelligible fashion and written in standard English??> Reviewer #2: Yes Reviewer #3: Yes ********** Reviewer #2: I found the comments from my last review to have been at least partially addressed. I do note that I strongly disagree with the authors’ statement in the response to reviewers that different statistical errors, topics or results would be “slight changes in the manipulation” and thus unlikely to change the observed effect. Statistical errors range from subtle to obvious, and their detectability (which is a prerequisite for taking them into account in the evaluation of an abstract) should be expected to vary accordingly. I do think that the one used in the vignette is on the subtle side, and I’d thus put much more emphasis on this limitation when discussing the results. But I think that this will be clear to the reader from the description of methods and results, so I won’t be fighting the authors further on this issue. That said, I have found a problem which I had overlooked in previous reviews, which I think is quite important and does have to be fixed before publication. Although the study was entirely conducted with psychology students and researchers, the fact that they come from this research field is nowhere to be found either in the title or abstract, which only mention “students” and “academics” with no information on their background. I’d thus strongly advise the editor and authors to mention this both in the title (e.g. “The effect of manuscript characteristics on quality assessment of scientific manuscripts in psychology”) and abstract (e.g. “…psychology academics give of articles submitted for publication, and psychology academics and students…”). Reviewer #3: Dear reviewers, based on the respective reviews that have been conducted. Please review the document carefully to ensure there are no duplicated points at the end of paragraphs, especially in the abstract. Please review the document again to verify that all the values mentioned are correctly described in the abstract. Please verify that the formatting is correct (spacing, punctuation, and consistency). We wish you the best of luck in advance for your contribution to the scientific community. ********** what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy Reviewer #2: Yes: Olavo B. Amaral Reviewer #3: No ********** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] To ensure your figures meet our technical requirements, please review our figure guidelines: https://journals.plos.org/plosone/s/figures You may also use PLOS’s free figure tool, NAAS, to help you prepare publication quality figures: https://journals.plos.org/plosone/s/figures#loc-tools-for-figure-preparation. NAAS will assess whether your figures meet our technical requirements by comparing each figure against our figure specifications. 13 Jul 2026 See attached file Submitted filename: response letter PLOS ONE revision2.docx 28 Jul 2026 The effect of manuscript characteristics on quality assessment of scientific manuscripts in psychology PONE-D-24-57377R3 Dear Dr. van Assen, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. For questions related to billing, please contact billing support. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Sonia Vasconcelos, PhD Academic Editor PLOS One Additional Editor Comments (optional): Reviewers' comments: PONE-D-24-57377R3 PLOS One Dear Dr. van Assen, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS One. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset You will receive further instructions from the production team, including instructions on how to review your proof when it is ready. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few days to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. You will receive an invoice from PLOS for your publication fee after your manuscript has reached the completed accept phase. If you receive an email requesting payment before acceptance or for any other service, this may be a phishing scheme. Learn how to identify phishing emails and protect your accounts at https://explore.plos.org/phishing. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS One and supporting open access. Kind regards, PLOS One Editorial Office Staff on behalf of Dr. Sonia Vasconcelos Academic Editor PLOS One