Does perceived test fairness affect test preparation? -- A case study of Duolingo EnglishTest Authors: Don Yao Published: 2024 ABSTRACT AbstractDrawing upon (Yao, 2023) [1]'s examination of perceived fairness in the Duolingo English Test (hereafter DET) and the crucial role of test preparation, the current study delves into the influence of perceived DET fairness on test preparation strategies. Guided by Kunnan et al. (2004) [2]'s Test Fairness Framework, a conceptual model encompassing two dimensions, i.e., perceived test fairness and test preparation practices, was formulated. Utilizing a validated DET Fairness and Preparation Questionnaire, data were gathered from 1012 Chinese test takers. Statistical analyses using SPSS v.26.0 revealed descriptive insights, indicating that test takers employed a diverse array of strategies for test preparation. Furthermore, EFA illuminated that eight factors of test preparation strategies were extracted (i.e., information searching, mass practice, training test-taking skills, social strategies, self-evaluation, test-related settings, affective strategies, and memory strategies). Subsequent CFA and SEM performed in AMOS v.26.0 further elucidated the causal relationships, underscoring the direct and significant impact of test takers' fairness perceptions on their test preparation strategies. This research offers a nuanced understanding of how perceived test fairness shapes test preparation strategies, thereby contributing to the field of language testing and language teaching and learning theoretically and practically.Keywords: Duolingo English test, Perceived test fairness, Test fairness framework, Test preparation Background Due to the COVID-19 pandemic, traditional in-person English proficiency tests, such as TOEFL and IELTS, were disrupted, leaving Chinese students seeking alternative methods to demonstrate their English language skills for admission to English-medium universities. One such alternative, the Duolingo English Test (DET), an online, at-home language proficiency test, gained popularity worldwide [3,4]. However, the DET has encountered doubts regarding its fairness, encompassing worries about security breaches, limited accessibility, the authenticity of test items, and possible adverse effects on educational outcomes [3,4]. In the realm of language testing and assessment, fairness is paramount, ensuring that a test or an assessment is equitable for all test takers and brings about positive outcomes [5]. Test preparation, an integral part of the learning process, is directly related to the test and plays a significant role in students' performance [6,7]. In the Chinese context, test preparation, defined as any intervention procedure specifically undertaken to improve test scores, whether by improving the skills measured by the test or by improving the skills for taking the test, or both [33], is a historically rooted and educationally acknowledged phenomenon, often presenting the most observable form of learning [8,9]. With the rise of the DET, Chinese students are now exploring various strategies to prepare for this new form of assessment. However, the question remains: how do they perceive the fairness of this online, at-home test, and how does it influence their preparation efforts? This emerging issue merits further investigation. [1]'s study on the subjective fairness of DET found that while test takers generally perceived the test as fair, the specific impact of test fairness on test preparation remains unclear. Based on the existing literature, it is evident that certain aspects of test fairness, as outlined in Ref. [2]'s Test Fairness Framework, notably validity, access, and impact, can significantly influence test preparation [10]. examined the relationship between test takers' perceptions of the validity of the College English Test Band 4 (CET-4) and their test preparation efforts. The study found that those who endorsed the test's validity were more likely to engage in test preparation, particularly cramming and drilling, which subsequently led to higher test scores. Similarly [11], focused on the influence of test access on test preparation, discovering that insufficient information about the test provided by universities negatively impacted test preparation. This highlights the importance of accessibility and transparency in ensuring a fair and effective test preparation process [12]. delved deeper into the effects of test validity, impact, and importance on various learning approaches, including test preparation. The study revealed that positive perceptions of test validity and impact encouraged students to engage more frequently in test preparation, which in turn enhanced their test scores. Collectively, these studies suggest that test fairness, particularly in terms of validity, access, and impact, plays a crucial role in shaping test takers' attitudes towards and engagement in test preparation. Therefore, it is essential to consider these factors when designing and administering language proficiency tests, such as the DET, to ensure a fair and effective assessment process that supports student learning and achievement. The prior studies on the effect of test fairness on test preparation, while insightful, have several limitations that restrict their comprehensiveness and generalizability [10].'s work, for instance, was confined to a low-tier city population with a limited range of English proficiency, thereby lacking diversity in its sample [11].'s study suffered from a small sample size of only six participants, rendering its findings less applicable to broader populations [12].'s study, meanwhile, relied on a limited number of questionnaire items to measure test preparation, potentially underestimating its complexity. To address these gaps, the current study aims to delve deeper into how Chinese test takers' perception of DET fairness affects their test preparation practices. By employing a multi-method approach combining interviews and a questionnaire survey, the study aims to both understand the nuances of test preparation through qualitative data and generalize the findings through quantitative analysis. Utilizing [13]'s coding scheme, the interviews will help uncover the diverse test preparation methods employed by test takers. Descriptive analysis will provide an initial understanding of the test preparation practices among this population. Further, EFA and CFA will be applied to the questionnaire data to ensure the reliability and validity of the measurement scales. Finally, SEM will be utilized to ascertain the intricate connections between perceived fairness in DET and test takers' test preparation practices. The results of this study have the potential to provide valuable theoretical and practical insights into the field of language assessment, especially regarding how subjective perceptions of test fairness shape test preparation behaviors. Given the significant number of Chinese test takers for the DET, these findings may have far-reaching implications for improving test design, administration, and ultimately, test takers' performance. Conceptual model and research questions To gain a more profound insight into how perceived test fairness impacts test preparation, the present research proposes a conceptual model grounded in Refs. [2,34]'s Test Fairness Framework (TFF), incorporating five primary qualities: validity, absence of bias, access, administration, and social consequences (see Table 1). This model comprises two interconnected components: perceived test fairness and test preparation. The model hypothesizes a direct relationship between test takers' perceptions of test fairness and their subsequent test preparation practices (see Fig. 1). In the context of this study, which focuses specifically on DET, the model aims to shed light on how Chinese test takers' subjective perceptions of the test's fairness directly shape their approaches to test preparation. To operationalize this model and test its hypotheses, two research questions (RQs) have been established. ●RQ 1: What strategies did test takers adopt when they prepared for the DET? RQ 1: What strategies did test takers adopt when they prepared for the DET? ●RQ 2: How do test takers' perceptions of DET fairness influence test preparation? RQ 2: How do test takers' perceptions of DET fairness influence test preparation? Hypothesized test preparation practices measurement model. Note. TPP= Test preparation practices; IS= Information searching; MP = Mass practice; TTS = Training test-taking skills; SS= Social strategies; SE= Self-evaluation; TS= Test-related settings; AS = Affective strategies; MS = Memory strategies. Table 1Test fairness framework (extracted from [2, 34]).Table 1Main Quality/Sub-qualitiesFocus/FociValidityContent representativeness/coverageRepresentativeness of items, tasks, topicsConstruct or theory-based validityConstruct/underlying traitCriterion-related validityScore comparison with external criteriaReliabilityStability, alternate form, internal consistency, inter-rater reliabilityAbsence of biasContent or language/dialectContent or language biasDisparate impactDifferential item functioningStandard settingCriterion setting and selection decisionsAccessEducationalOpportunity to learnFinancialAffordabilityGeographicalLocation and distancePersonalAccommodationsEquipment and conditionsFamiliarityAdministrationPhysical settingPhysical conditionsUniformityUniformity of administrationSecurityFraud, misrepresentation, cheatingSocial consequencesWashbackEffect on instructionRemediesRe-scoring, re-evaluation, legal remedies Test fairness framework (extracted from [2, 34]). Conceptual model. Methods Participants For the interview segment of the study, a total of 12 participants (consisting of six males and six females) were recruited. Additionally, 1012 participants (498 males and 514 females) participated in the questionnaire survey. All participants were Chinese university students aged 21 to 24, with Chinese as their native language and English as their second language. They had at least 12 years of English learning experience and had prepared for the DET for approximately six weeks on average. On average, they had taken the DET twice. Their English proficiency, indicated by their DET scores, ranged mostly between 85 and 140, with only a handful scoring 80 or below. This diversity in proficiency levels offers a comprehensive dataset for examining the correlation between perceived test fairness and test preparation practices. Instrument Interview protocol The interview protocol comprises two distinct sections. The initial section seeks the participants' consent to participate in the study, establishing a foundation of trust and mutual understanding. The subsequent section follows the interview question design framework proposed by Ref. [14]. This section begins with opening questions designed to establish rapport with the participants, such as inquiring about the duration and frequency of their DET preparation. The main section of the protocol then delves deeper into the participants' test preparation experiences. It commences with general inquiries about their preparation practices, gradually progressing to more specific and clarifying questions aimed at understanding the specific methods or strategies they employed during this phase. The use of what, why, and how questions facilitates a comprehensive exploration of their experiences and perspectives. Finally, the protocol concludes with an open-ended question, inviting the participants to share any additional insights or perspectives they may have regarding test preparation. This allows them to have a final say and ensures that all relevant aspects of their experiences are captured. The designed interview protocol underwent rigorous review and approval by the thesis advisory committee (TAC), indicating its validity and adequacy in addressing the research objectives. This approval ensures that the questions are appropriately framed to elicit meaningful and insightful responses from the participants. DET Fairness and Preparation Questionnaire In light of [2,34]'s Test Fairness Framework (TFF) and interview results (see Appendix 1), a DET Fairness and Preparation Questionnaire was crafted to assess participants' perceptions of test fairness and their test preparation practices. Initially, the questionnaire underwent a pilot test with 109 participants, resulting in an overall reliability value of .739. Following this pilot, two items were refined to enhance clarity and relevance, leading to the final and refined version. The questionnaire is structured into three sections. Section A collects demographic details including age, gender, region, and DET scores. Section B, which consists of 25 items, delves into participants' perceptions of test fairness, addressing aspects such as validity, absence of bias, accessibility, administration, and societal impacts. Section C, encompassing 64 items, explores participants' test preparation practices, covering areas like information seeking, general strategies, social approaches, self-assessment, test-related environments, emotional strategies, and techniques for enhancing English skills. All items are rated on a six-point Likert scale, ranging from 1 (strongly disagree) to 6 (strongly agree), allowing participants to select the option that best reflects their views. The questionnaire is available in both English and Chinese to cater to participants' language preferences. To ensure the internal consistency and reliability of the finalized questionnaire, a Cronbach's Alpha coefficient was calculated. The result, with an overall value of .807, indicates a high level of reliability, thus validating the questionnaire for further analyses. Data collection For the interview portion, participants were interviewed individually by the researcher at their preferred time, with interview durations ranging from 18 to 27 min. The interviews were conducted via phone calls and were audio-recorded to ensure accuracy. To promote optimal communication and understanding, participants were given the option to choose between English and Chinese Mandarin, and all participants opted for Chinese Mandarin. The semi-structured format of the interviews allowed for flexibility, encouraging participants to offer their insights and perspectives freely at any point. All interview data were securely stored on the researcher's personal cell phone and laptop, with each participant's data labeled sequentially from S1 to S12. For the questionnaire survey, the online platform Wenjuanxing was utilized for questionnaire distribution. Prior to distribution, the questionnaire's face validity was verified to ensure its relevance and appropriateness. At the outset, the questionnaire clearly stated its main objectives, reassuring participants that their responses would be kept confidential and used solely for academic purposes. Additionally, all participants provided informed written consent, which was included on the first page of the questionnaire. While there was no strict time limit for participants, all questions were designated as mandatory to prevent missing data in subsequent analyses. Data analysis The qualitative interview data underwent a rigorous transcription process using the open platform iFLYTEK (https://global.xfyun.cn/). The verbatim transcripts were then converted into Word files. Once the accuracy of the transcripts was verified, the data were imported into NVivo v.13.0 for in-depth analysis. To ensure consistency and reliability, two researchers independently applied [13]'s coding scheme. Prior to independent coding, a norming session was conducted [15] to align understanding and expectations. Following this, they independently coded the remaining interview transcripts. Afterward, they convened to discuss any discrepancies in coding and reached a final consensus, ensuring the robustness and reliability of the coding process. A multifaceted analytical approach was employed to examine the questionnaire survey data. Initially, descriptive statistics were applied to gain a general perspective on test-takers' perceptions of DET fairness and their test preparation practices. For a deeper understanding, factor analysis was conducted using SPSS v.26.0 and AMOS v.26.0. Specifically, EFA was performed at the item level to investigate whether items intended to measure similar constructs clustered together. Following this, CFA was utilized at the sub-scale level to evaluate the reliability and internal validity of each measurement model. Ultimately, SEM was applied to model the complex relationships between perceived DET fairness and test preparation practices. This comprehensive analytical approach provided a solid understanding of the data. Results and discussion ● RQ 1: What strategies did test takers adopt when they prepared for the DET? RQ 1: What strategies did test takers adopt when they prepared for the DET? Based on the in-depth interviews conducted with 12 participants, the researcher identified seven preliminary categories that emerged from the data. These categories, namely information searching, social strategies, affective strategies, test-related settings, self-evaluation, general methods, and English abilities improvement, were synthesized through careful analysis of the participants' responses. For each category, representative codes were developed to encapsulate the key themes and patterns within the interviews. The researcher has summarized these preliminary categories and corresponding codes in Tables 0–1, which is included in Appendix 1 for reference. Descriptive analysis Table 2 presents the descriptive statistics of participants' test preparation practices for the DET. Notably, the skewness and kurtosis values for the dataset fall within the range of −2 to +2, indicating a normal distribution. This normal distribution allows for the application of parametric statistical tests for further analysis. The mean score for test preparation practices is 3.823, with a standard deviation of 1.097, suggesting that participants employ a wide range of methods and strategies to prepare for the DET. This finding aligns with the common experience of Chinese students, who often adopt diverse approaches in preparing for tests throughout their academic journey from kindergarten to university. Table 2Descriptive results of test preparation practices (N = 1012).Table 2Item No.MeanStd. DeviationSkewnessKurtosisInformation searching3.6371.065.490−.357D423.5881.018.284−.115D433.5121.066.333−.031D443.6651.042.309.026D453.7821.081−.196−.313General methods3.6251.101−.224−1.072D463.5071.030−.199−.998D473.6231.046−.128−1.407D483.5401.134−.239−1.372D493.4971.158−.122−1.439D503.9011.167−.276−1.001D513.5101.149.137−.979D523.4851.192−.152−1.372D533.9031.015−.124−1.379D543.6561.020−.189−1.361Social strategies3.6001.199−.396−1.384D553.4451.004.506−1.436D563.7681.032−.120−1.388D573.6781.033−.163−1.377D583.5091.199−.235−.403Self-evaluation3.6881.152−.504−.446D593.5071.185−.235−.366D603.6561.168−.223−.412D613.8491.126−.475−.426D623.8781.159−.252−.292D633.4591.142−.258−.306D643.7781.190−.250−.353Test-related settings3.6461.231−.198−.449D653.8761.166−.256−.323D663.4561.168−.280−.298D673.5481.025.169−.866D683.5021.336.149−.941D693.5301.003.165−.830D703.8871.128.237−.828D713.8781.040.414−.142D723.4941.281.471.043Affective strategies3.8851.138.361−1.121D733.9421.281.4581.138D743.9271.283.451.966D753.7871.096.5161.048D763.8761.012.354−.962D773.8791.180.325−.371D783.9011.073.260−.799Improving L abilities3.5811.071.336−.380D793.6751.178.380−.279D803.6711.127.328−.241D813.7191.199.420−.037D823.5421.106.317−.423D833.3981.103−.514.997D843.4791.022−.298.483Improving S abilities3.5111.195−1.2571.411D853.8781.036−1.022.834D863.7111.018−1.0021.334D873.9871.191−.988.544D883.6771.098−.799.835D892.7821.099−.8781.467D902.9861.015−1.015.901D913.5771.092−.377.951D923.489.945−.466.747Improving R abilities4.5471.189.7661.212D935.337.992.5231.223D945.225.999.3391.657D955.0781.0761.0071.601D963.9871.015−.426.675D973.7791.057−.492−.664D983.8781.099−.357−.682Improving W abilities4.5161.089.998−1.025D995.2251.1211.1231.878D1004.8761.008.978−.989D1013.9901.076−.977−1.221D1024.9871.045−.8121.657D1034.0121.323−1.090−.654D1045.332.778−.9561.102D1053.189.937−.836−.989Overall3.8231.0971.0231.387Note. No. = Number; Std. Deviation = Standard Deviation; L = Listening; S= Speaking; R= Reading; W= Writing. Descriptive results of test preparation practices (N = 1012). Note. No. = Number; Std. Deviation = Standard Deviation; L = Listening; S= Speaking; R= Reading; W= Writing. The table highlights the prominent test preparation practices among the participants, with a focus on affective strategies and training test-taking skills. For affective strategies, participants scored relatively high (M = 3.885; S.D. = 1.138), indicating that they actively sought to maintain a positive and relaxed mindset while regulating their emotions during preparation. This aligns with [16]'s findings, which suggest that such strategies can positively impact English language acquisition. In terms of training test-taking skills, participants also scored notably (M = 3.625; S.D. = 1.101), with a particular emphasis on improving speaking abilities through imitation of English pronunciation in movies and news. This is understandable given the Read-aloud task in the DET, which requires accurate pronunciation within a limited time frame. The effectiveness of imitation in enhancing English speaking skills is supported by Ref. [17]'s research. Furthermore, the accessibility of English movies and news resources has facilitated this learning approach. Additionally, the use of automated writing evaluation systems, such as the Pigai system, is prevalent among Chinese students, with over 1000 universities incorporating it into their English writing classes [18]. These systems provide valuable feedback and scores to help students refine their writing skills, albeit with some room for improvement in precision [18]. emphasizes the importance of instructor guidance in conjunction with these systems to maximize their effectiveness. Overall, these findings demonstrate the diversity of strategies employed by test takers to prepare for the DET, with a strong focus on both affective and skill-based approaches to language learning. Next, the analysis reveals that test takers utilized various other strategies in their preparation. Social strategies (M = 3.600; S.D. = 1.199) were prevalent, as participants sought help from peers, teachers, or native speakers when encountering difficulties with the test. This aligns with [19]'s emphasis on the importance of social strategies in L2 learning processes. Test takers also engaged in information searching (M = 3.637; S.D. = 1.065) to gather resources and materials relevant to the test. In addition, they paid attention to test-related settings (M = 3.646; S.D. = 1.231), such as finding a quiet environment for review, training typing speed, and practicing focusing on the camera. These preparations are particularly crucial for online and at-home tests, as the DET has strict rules to ensure exam security. For example, if there is noise during the test or the test taker's eyes are not focused on the camera for 3 s, it may be considered cheating. Moreover, without scratch paper, test takers may need to adapt their test-taking skills to this new format. Also, test takers utilized self-evaluation strategies (M = 3.688; S.D. = 1.152) to assess their progress and identify areas for improvement. The current study highlights the importance of considering these additional strategies and settings, especially for online and at-home tests. These factors, which may be considered construct-irrelevant variables, can significantly influence test takers' performance. Therefore, practitioners examining such tests should be mindful of their potential impact on test fairness and ensure that all test takers have equal opportunities to demonstrate their abilities. Finally, for self-evaluation, test takers exhibited a proactive approach by analyzing score distributions and assessing the relative importance of different sections. This strategy is effective as it enhances their ability to dissect test papers and identify effective preparation methods. Through this process, they gain insight into their strengths and weaknesses, enabling them to focus on improving their weaker areas. Additionally, test takers expressed interest in understanding the mechanism of computer scoring. While the interviews revealed that they lacked a deep understanding of the rationale behind machine scoring, they nonetheless attempted to deduce strategies for achieving higher scores. Given that the DET relies entirely on automated scoring, and the scoring algorithms are not publicly disclosed, this creates a "black box" effect, as [3] observed. To enhance transparency and fairness, it is recommended that the DET developers consider making the scoring algorithms or rating rubrics publicly available. This would empower test takers with a clearer understanding of the aspects they need to improve and how their performance is evaluated. Such transparency would also strengthen the credibility and trustworthiness of the testing system. ●RQ2: How do test takers' perceptions of DET fairness influence test preparation? RQ2: How do test takers' perceptions of DET fairness influence test preparation? EFA results First, EFA was conducted at the item level to assess whether items designed to measure the same underlying construct grouped together. A random selection of 337 participants' questionnaire responses was utilized for this purpose. TheKaiser-Meyer-Olkin (KMO) value for the perceived test fairness construct was found to be .823, exceeding the threshold of .60 [20]. Additionally, Bartlett's test of sphericity confirmed the suitability of the dataset for factor analysis, with a significant Chi-square value of 10477.400 (p < .001). These findings indicate that the dataset is appropriate for further analysis. The communality results of test preparation practices reveal that nine values are below .30 [21]: D51 (h2 = .266), D52 (h2 = .182), D60 (h2 = .288), D63 (h2 = .255), D72 (h2 = .274), D76 (h2 = .269), D77 (h2 = .105), D82 (h2 = .146), and D83 (h2 = .148). Consequently, these items are excluded, leaving 55 items for further Confirmatory Factor Analysis (CFA). The total variance results for test preparation practices reveal 14 factors with eigenvalues exceeding 1.00 [20,32], accounting for 72.012 % of the total variance of the remaining 55 indicators. The scree plot further supports this fourteen-factor solution, as the sharp break occurs after the fifteenth factor. Therefore, based on both the total variance and scree plot results, a total of 14 factors are retained. Finally, the structure matrix results of test preparation practices also note a fourteen-factor structure, and 55 items are categorized into 14 factors. Almost all factor loadings are above .50 except D62 with the value of .497. Though it is a little lower than the standard [22], as the discrepancy is not too large, the item is regarded as acceptable. Fourteen factors are renamed (F1: Mass practice test papers; F2: Training general test-taking skills; F3: Reading memory strategies; F4: Information searching; F5: Test-related settings; F6: Affective strategies; F7: Self-evaluation; F8: Mass practice reading abilities; F9: Mass practice listening abilities; F10: Mass practice speaking abilities; F11: Social strategies; F12: Training writing test-taking skills; F13: Writing memory strategies; F14: Training speaking test-taking skills), and they are recategorized into eight intended scales (i.e., information searching, mass practice, training test-taking skills, social strategies, self-evaluation, test-related settings, affective strategies, and memory strategies). Five scales are existing names from the questionnaire, and the other three scales (i.e., mass practice, training test-taking skills, and memory strategies) are referenced from Ref. [10]'s Test Preparation Questionnaire and [19]'s Strategy Inventory for Language Learning. Mass practice means test takers practicing a lot of test items to improve their test performance. Training test-taking skills represent test takers using different methods to train their skills that are useful for taking the test. Memory strategies refer to “creating mental linkages, applying images and sounds, reviewing well, and employing actions” [19]. Table 3 presents all 14 factors and their intended scales. Factors are grouped together if they carry similar meanings, and their reliability is examined. The Cronbach's Alpha coefficients vary from .682 to .900, indicating the internal consistency of the items is high, and these items are reliable. Though .682 is a little lower than .70 [23,24], as it is pretty approaching the .70 point, and [25] argue that “.60 to .70 indicates an acceptable level of reliability” (p. 681), the coefficient is considered acceptable. Table 3Factor summarization of test preparation practices.Table 3Intended scale(s)Factor(s)Item(s) No. of Item(s) Cronbach's. Alpha (α)Information searchingF4: Information searchingD42, D43, D44, D454.728Mass practiceF1: Mass practice test papersF10: Mass practice speaking abilitiesF9: Mass practice listening abilitiesF8: Mass practice reading abilitiesD46, D47, D48, D49D89, D90, D91, D92D79, D80, D81, D84D96, D97, D984443.843Training test-taking skillsF2: Training general test-taking skillsF14: Training speaking test-taking skillsF12: Training writing test-taking skillsD50, D53, D54D85, D86, D87, D88D100, D101, D103, D105344.786Social strategiesF11: Social strategiesD55, D56, D57, D584.767Self-evaluationF7: Self-evaluationD59, D61, D62, D644.682Test-related settingsF5: Test-related settingsD65, D66, D67, D68, D69, D70, D717.724Affective strategiesF6: Affective strategiesD73, D74, D75, D784.708Memory strategiesF3: Reading memory strategiesF13: Writing memory strategiesD93, D94, D95D99, D102, D10433.900Note. F= Factor; No. = Number. Factor summarization of test preparation practices. Note. F= Factor; No. = Number. CFA results At the sub-scale level, CFA was subsequently examined to evaluate the reliability and internal validity of each measurement model, utilizing the responses from the remaining 675 participants. A model was hypothesized, with test preparation practices comprising 1 s-order latent factor and eight first-order latent factors. The second-order factor represents overall test preparation practices, while the eight first-order latent factors, consisting of 55 items, encompass information searching, mass practice, training in test-taking skills, social strategies, self-evaluation, test-related settings, affective strategies, and memory strategies. To be noted, when the scales contain combined items, meaning that items are from different factors, original item values are used in the sums of variables to ensure the averages of the scales are comparable, and the items are measured on the same scales from 1 to 6 [10]. Corresponding scales are mass practice (with the combination of mass practice test papers, mass practice speaking abilities, mass practice listening abilities, and mass practice reading abilities), training test-taking skills (with the combination of training general test-taking skills, training speaking test-taking skills, and training writing test-taking skills), and memory strategies (with the combination of reading memory strategies and writing memory strategies). The assessment of multivariate normality for test preparation practices indicates a kurtosis value of 590.694, exceeding the threshold of 10 as recommended by Refs. [26,31]. Consequently, the assumption of normal distribution is not met, prompting the use of the bootstrapping method in subsequent analyses. Additionally, the model fit indices suggest that the hypothesized model aligns reasonably well with the data (CMIN/DF = 2.136, RMSEA = .048, CFI = .875, SRMR = .044), falling within the acceptable ranges (CMIN/DF < 3, RMSEA <.08, CFI >.90, SRMR <.08) as per [[27], [28], [29]] (see Table 4). Although the CFI value slightly misses the .90 mark, it is still considered acceptable given that [30] suggests a value of .80 as sufficient, thereby deeming the model admissible. Table 5Standardized path coefficients results.Table 5Path(s)SPC (β)S.E.C.R.pTPP<---PF.381.0576.669∗∗∗IS<---TPP.738.03322.865∗∗∗MP<---TPP.834.06812.334∗∗∗TTS<---TPP.778.05613.785∗∗∗SS<---TPP.830.04618.076∗∗∗SE<---TPP.723.03819.741∗∗∗TS<---TPP.888.03823.989∗∗∗AS<---TPP.772.02432.876∗∗∗MS<---TPP.749.04018.200∗∗∗Note. ∗∗∗ represents p < .001; ∗∗ represents p < .01; ∗ represents p < .05; SPC= Standardized path coefficients (β); S.E. = Standardized error; C.R. = Critical ratio; TPP= Test preparation practices; IS= Information searching; MP = Mass practice; TTS = Training test-taking skills; SS= Social strategies; SE= Self-evaluation; TS= Test-related settings; AS = Affective strategies; MS = Memory strategies; TP= Test performance. Standardized path coefficients results. Note. ∗∗∗ represents p < .001; ∗∗ represents p < .01; ∗ represents p < .05; SPC= Standardized path coefficients (β); S.E. = Standardized error; C.R. = Critical ratio; TPP= Test preparation practices; IS= Information searching; MP = Mass practice; TTS = Training test-taking skills; SS= Social strategies; SE= Self-evaluation; TS= Test-related settings; AS = Affective strategies; MS = Memory strategies; TP= Test performance. Table 4Model fit indices of perceived test fairness measurement model.Table 4ModelCMIN/DFRMSEACFISRMRFig. 22.316.048.875.044Note. CMIN/DF = Minimum discrepancy per degree of freedom; RMSEA = Root mean squared error approximation; CFI= Comparative fit index; SRMR= Standardized root mean squared residual. Model fit indices of perceived test fairness measurement model. Note. CMIN/DF = Minimum discrepancy per degree of freedom; RMSEA = Root mean squared error approximation; CFI= Comparative fit index; SRMR= Standardized root mean squared residual. SEM results SEM was ultimately employed to establish a model depicting the hypothesized relationships between perceived DET fairness and test preparation. The results demonstrated a reasonable fit for the model (CMIN/DF = 2.159, RMSEA = .052, CFI = .891, SRMR = .045). Although the CFI value slightly falls below the preferred threshold, it is still deemed acceptable due to its proximity to .90 and given the complexity of the model. Fig. 3 presents the final structural equation model, including standardized regression weights. Final Structural Equation Model with Standardized Parameters Furthermore, as shown in Table 5, test takers' perceptions of DET fairness exhibit a statistically significant positive correlation with several factors: perceived validity (β = .808, p < .001), perceived absence of bias (β = .831, p < .001), perceived access (β = .773, p < .001), perceived administration (β = .819, p < .001), and perceived social consequences (β = .770, p < .001). Given that the previous section only explained the overall effect of perceived test fairness on test preparation, specific effects, i.e., the effect of sub-dimensions of perceived test fairness on test preparation, were further examined to better understand the different associations. Specifically, test takers' perceptions of validity had a positive effect on test preparation, indicating that test takers who endorsed more in test validity tended to engage more in test preparation. This result is similar to Ref. [10]'s and [12]'s studies with the conclusion that test takers' perceptions of CET-4 validity and NMET validity positively influenced their test preparation, respectively. According to Ref. [10], engaging in the practice of valid test papers can serve as a sensible approach to improving test performance and language proficiency. Furthermore, test takers' perceptions of the absence of bias had a positive impact on their test preparation efforts, indicating that those who believed more strongly in the lack of bias tended to invest more in preparing for the test. Consequently, it is crucial for test developers to uphold the quality of test items to foster such perceptions among test takers. In addition, test takers' perceptions of access to testing positively influenced their preparation efforts, as those who valued greater access tended to engage more in preparing for the test. This finding is plausible given the various resources available to students today. Educationally, students can download practice DET test papers from the official website and access numerous websites offering both real and mock test papers. Financially, the DET, costing 49 U S. dollars, is more affordable than other standardized tests like IELTS and TOEFL, which cost around 300 U S. dollars in China. Geographically, students can take the DET anywhere, saving time, energy, and expense, allowing them to focus more on preparation. In terms of conditions and equipment, students can use their own devices to take the DET, and the test procedures are clear to each test taker, aiding in better preparation. While IELTS and TOEFL also offer an online mode, test takers must use computers in designated test rooms. However [11],'s study found that universities provided limited test-related information, leading students to seek help from private institutions. This suggests that universities should provide sufficient test-related information to ensure all students have equal opportunities to prepare based on the information available, as seeking private help relies heavily on family financial abilities. Furthermore, test takers' perceptions of the administration of the DET positively influenced their test preparation. Specifically, those who valued the administration of the test, which is particularly crucial since it is entirely conducted at home, tended to engage more in preparation. According to Ref. [4], the security measures for the DET are rigorous, with high technological demands to ensure its integrity. These include requirements such as maintaining eye contact with the camera, prohibiting the use of headphones and mobile phones, and preventing the browser from being exited during the test. Additionally, students need to consider the physical conditions of the room, such as lighting and temperature, to meet the invigilation standards. These factors suggest that test takers who recognized the importance of test administration were more likely to prepare thoroughly to avoid any accusations of cheating. Lastly, test takers' perceptions of social consequences had a positive impact on their test preparation. Those who placed greater significance on social consequences tended to invest more effort in preparing for the test. This trend may be attributed to the fact that many test takers undertake the DET for the purpose of university admissions. With the understanding that the test can aid their English learning, they may be more motivated to prepare diligently in order to achieve a higher score. Conclusion The current study delved into the impact of perceived DET fairness on test preparation, utilizing SEM analysis. The findings revealed that test takers employed diverse strategies for preparation, and their perception of DET fairness positively influenced their preparation efforts. The study offers significant implications for both theory and practice. Theoretically, this research introduced a conceptual model that builds upon [2,34]'s Test Fairness Framework (TFF), demonstrating its flexibility in examining the influence of test takers' perceptions on their preparation behaviors. Additionally, SEM was developed to clarify the relationship between test takers’ perceived fairness of DET and their test preparation. Practically, this study can guide test takers in enhancing their understanding of effective test preparation strategies and the aspects to consider for a fully remote test. However, it is crucial to acknowledge some limitations. Despite the diversity in participant demographics, a multi-level analysis of the data was not implemented. Therefore, future research endeavors should incorporate additional demographic factors, including gender, region, and English proficiency, to attain a more holistic understanding. In summary, this study underscores the significance of subjective test fairness as a vibrant and crucial area of research within language assessment, as well as second language teaching and learning. It is anticipated that future researchers will persist in enhancing studies in this vital domain. Data availability statement There are no any data presented in this research. Funding This research was funded by Zhejiang Federation of Humanities and Social Sciences Circles (No: 24NDQN206YBM), and the project topic is Investigating the effect of ChatGPT on L2 education. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Interview Results Tables 0–1 showcases seven preliminary categories along with their respective representative codes, which have been meticulously summarized by the researcher. It is important to note that only those codes mentioned by more than three participants have been listed below. The category of Information Searching encompasses test-takers who utilize diverse methods to seek out information beneficial for their examination. Primarily, they search for DET test papers to gain more practice opportunities. The majority of them rely on the website DengDeng Duolingo (https://det.91ddedu.com/#/), which offers a comprehensive suite of resources. Six participants clearly stated in the interview. S2, S4, S5, S7, S8, S9:It is a relatively comprehensive website that contains different difficulty levels and section types of test items. I can also screen what I have done before, and what I have not done yet. Additionally, four participants sourced test preparation and test-taking strategies from popular Chinese information-seeking websites and platforms such as Little Redbook, Zhihu, and Microblog. These resources enabled them to prepare for the test more systematically and tackle test items more efficiently. Furthermore, three participants actively monitored the Duolingo official website and WeChat public account to stay informed about the latest test-related updates. When it comes to social strategies, as Oxford (1990) eloquently put it, they entail “asking questions, collaborating with others, and understanding their viewpoints" (p. 247). The research highlights that four participants don't hesitate to reach out for help when faced with test-related hurdles, and they eagerly seek input on their English communication. Furthermore, three participants shared that they rely on the insights of their peers who have taken the DET test, English teachers, or family members for advice and encouragement. One participant further explained. S2:It was easy to get lost when preparing for the test. After finishing some practice test papers, I checked the answers by myself. If there was no improvement at all, I became quite worried. That is why I would ask others for help. Sometimes, their experience and suggestions helped me a lot. Affective strategies, in alignment with Oxford's (1990) perspective, are characterized as techniques that alleviate anxiety, foster self-encouragement, and monitor emotional well-being (p. 247). During the test preparation phase, five students reported maintaining a positive and relaxed mindset while effectively managing their emotions. Additionally, four participants emphasized their efforts to bolster their confidence. Upon further inquiry by the researcher about their confidence-building strategies, one student shared. S11:Whenever I made any progress, I would buy myself beautiful clothes or have meals with my friends. This kind of happiness was, therefore, doubled. For one thing, I made progress as the correctness of answering questions is higher; for another, I had new clothes and my stomach received satisfaction. The theme of test-related settings is particularly novel given that DET stands as one of the pioneering online and at-home English proficiency tests. Consequently, traditional test preparation methods may not be entirely applicable to DET. In preparation, six participants replicated the actual test environment, seeking out quiet spaces for review and creating conducive conditions by turning off their mobile phones and adjusting indoor temperatures. Additionally, four participants honed their typing skills and became accustomed to digital testing formats. Two participants mentioned. S12:My typing speed was quite slow as I did not receive any training. If you asked me to type on my phone, of course, I could type very fast. But typing on the computer was quite different. I remember I used around two weeks to train my typing speed. But it was not bad, because it represents that I also practiced a lot of compositions.S1:This was actually the first time for me to take an online test. This means there was no paper given in the test. You know, I was accustomed to writing or marking some important information on paper. At the beginning of preparing for DET, I indeed suffered a lot. Besides, four participants trained their eyes to focus on the camera and trained themselves to control the mouse within the movable range. One participant explained. S7:I took the DET twice. But you would not believe why I failed my first test. I just felt ridiculous because the system determined that I cheated in the exam as my eyes did not focus on the camera. Sometimes when people were stressed out, they did not know where the particular letter was. They may not be able to look at the camera all the time. So, when I took the test for a second time, I tried to look at the camera all the time. Self-evaluation, as highlighted in Xie's (2010) research, involves test-takers employing various techniques to assess, analyze, and reflect on their performance in order to make advancements during the preparation phase. Our study found that eight participants consistently analyzed their performance after each mock test, summarizing their mistakes and devising strategies to avoid similar errors in future encounters. Three participants delved deeper into score distributions, paying particular attention to the initial task types of DET, namely Yes or No Vocabulary Tasks (both Aural and Visual), due to the test's computer-adaptive nature. High scores in these initial tasks often lead to more challenging subsequent tasks and potentially higher overall scores. Conversely, encountering progressively easier tasks may serve as a less favorable indicator. Furthermore, four participants actively explored the mechanics of computer scoring. One participant had an interpretation. S6:I did not know whether my understanding was correct or not. But I found the DET scoring system may not be able to take contextual information into consideration. For example, I wrote good into god once. The meaning of the sentence was totally different, but I still got a high mark on the test. Also, as long as using some high-level academic words in your composition, your mark would not be very low. General methods encompass the commonly employed strategies utilized by participants. Throughout the preparation stage, all participants engaged in numerous test items. Some tailored their practice according to the level of difficulty or section types, while others focused on test items carrying a higher proportion of scores. Additionally, four participants devised a study schedule and reviewed the test accordingly, and another four participants consulted DET coaching books to deepen their understanding of the exam. With the comprehensive platform, featuring a continually updated and fully-covered practice question bank, instant scoring, and a fully authentic mock test experience, participants can enhance their preparation even further. Our mobile-friendly practice app allows for anytime, anywhere learning, while detailed performance analytics and customized learning plans cater to individual needs, ensuring precise score improvement. Enhancing English proficiency encompasses refining listening, speaking, reading, and writing skills. To bolster listening comprehension, four participants repeatedly listened to DET audio materials to become accustomed to pronunciation and intonation, while eight participants diligently listened to unfamiliar audios until they fully understood them. For speaking skills, seven participants watched pronunciation tutorials and mimicked the English accents. Three participants recorded their speech to scrutinize logic and grammar, while another three practiced in front of a mirror to observe their facial expressions and maintain focus. Ten participants crafted templates tailored for the speaking section. In terms of reading, all participants extensively memorized vocabulary and noted down unfamiliar words. Some even went beyond basic memorization by reciting word parts, roots, and prefixes/suffixes. For writing, all participants developed templates for the writing section. Four participants utilized automated writing evaluation systems to assess their writing quality and received feedback-driven guidance. Additionally, six participants memorized sophisticated vocabulary and well-crafted sentences to enrich their essays. Notably, most participants leaned towards traditional methods such as vocabulary memorization and template creation. To gain insights, the researcher randomly selected two participants and inquired about their choice of methods. Their reasons were quite similar. S9, S10:Before taking the test, I hoped I could have something in mind. Otherwise, everything was uncertain to me and I would feel extremely stressed out. If I memorized some vocabulary or prepared some speaking and writing templates, I could not say how much my mark would be improved, but I would feel more comfortable. Drawing from the codes derived from interview results, we developed a comprehensive list of 68 test preparation items. Illustratively, the first item corresponds to the code "search for past and practice DET test papers," phrased as "During my DET preparation, I searched for past and practice DET test papers." Similarly, the second item aligns with the code "search for test preparation and test-taking methods," stated as "While preparing for the DET, I searched for test preparation and test-taking methods." Notably, twenty of these items were inspired by Xie's (2010) Test Preparation Questionnaire and Oxford's (1990) Strategy Inventory for Language Learning, as their constructs resonate with the objectives of current study. Initially, based on the interview findings, the researcher tentatively categorized these items into seven categories: information searching, social strategies, affective strategies, test-related settings, self-evaluation, general methods, and improving English abilities. To gain a deeper understanding, further Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) will be conducted to ascertain the factors underlying test preparation practices and scrutinize their internal structures. Tables 0–1Categories and Codes from Interview ResultsTables 0–1Category(-ies)Code(s) No. of ParticipantsInformation searchingSearch for past and practice DET test papers6Search for test preparation and test-taking methods 4Search the latest test-related information3Social strategiesask how to improve DET performance3ask for help when meeting any problems4consult test review experience3point out my problems while talking in English4Affective strategiesmaintain a positive and relaxed state5regulate my emotions5build up my confidence4Test-related settingsimitate the actual test scene6train typing speed4train paperless tests4train my eyes to focus on the camera6train myself to control the mouse6Self-evaluationsummarize the mistakes made in practice tests8analyze test score distributions3analyze my performance8explore the mechanism of computer scoring4General methodspractice a lot of test items12practice test items based on the level of difficulty11practice test items based on section types11pay attention to the test items with a high proportion of scores6attend English training classes4make a schedule4Improving English abilitiesListeningSpeakingReadingfamiliar with pronunciation and intonationrepeatedly listen to unfamiliar audiosimitate the English pronunciationwatch pronunciation tutorial videosrecord myself speakingpractice in front of the mirrorcreate a templatememorize a lot of vocabulary4877331012Writingnote down the vocabulary that I do not mastercreate a templatememorize advanced-level words12126use automated writing evaluation systemsmemorize well-written sentences46 Categories and Codes from Interview Results References Associated Data Data Availability Statement There are no any data presented in this research.