Development and validation of a questionnaire to assess knowledge, attitudes, and practices regarding the addictive and detrimental effects of novel psychoactive substances among university students Authors: Liqin Lu, Jue Deng, Fenglan Li, Xiaoli Zhu Published: 2024 ABSTRACT AbstractBackgroundThe misuse of novel psychoactive substances (NPSs) is highly prevalent among university students, leading to significant public health and social safety concerns. To improve our drug education efforts related to NPSs, we developed a knowledge, attitudes and practices (KAP)-based questionnaire for university students.MethodsWe developed a KAP-based questionnaire after performing a comprehensive literature review and consulting experts. The questionnaire was then administered to randomly selected college students in Fujian Province. The items were evaluated using the critical ratio, correlation analysis, and Cronbach’s α coefficient. Furthermore, exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were used to assess the construct validity of the questionnaire. Subsequently, the convergent validity, discriminant validity, and reliability of the questionnaire were evaluated.ResultsA total of 587 college students were enrolled, including 280 (47.7%) males and 550 (93.7%) students who had received anti-drug education. Item analysis revealed that 3 items did not meet the t value (t < 4) or correlation coefficient thresholds (r < 3) or that Cronbach’s α coefficient increased after removing these items. After conducting EFA, 7 items with multiple loading factors above 0.4 or loading factors below 0.4 were removed, and 5 subdimensions were observed. CFA indicated that the questionnaire model exhibited a strong fit and acceptable convergent validity. However, there were some limitations in discriminant validity. The Cronbach’s α coefficients for the overall questionnaire and each dimension exceeded 0.9, indicating excellent internal consistency. Additionally, the questionnaire exhibited excellent split-half reliability (Cronbach’s α coefficients > 0.8, Spearman-Brown value = 0.982).ConclusionThe questionnaire developed herein exhibited high levels of reliability and validity, making it a suitable KAP-based tool for evaluating NPS-related addiction and harm among university students. These findings can guide the development of drug education and behavioral interventions.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12889-024-20458-0.Keywords: Novel psychoactive substance (NPS), Addictive and detrimental effects, University students, Reliability and validity, Knowledge attitudes and practices (KAP) Background Novel psychoactive substances (NPSs) are a category of newly developed narcotic or psychoactive compounds that are not regulated under the 1961 Single Convention on Narcotic Drugs or the 1971 Convention on Psychotropic Substances [1]. Despite the absence of regulation, these substances pose health risks comparable to those of substances listed under these conventions. According to the 2022 World Drug Report, 1,127 types of NPSs were identified globally, which was three times greater than the number reported in the previous year [2]. This sudden surge in NPSs presents a significant challenge to public health and drug control. In China, there has been a gradual increase in the abuse of NPSs, which has become a primary focus for public security departments. In addition to the psychoactive effects that induce euphoria and hallucinations, various harmful physical consequences may result from the misuse of NPSs [3]. For example, misuse of synthetic cannabinoids could result in vomiting, cognitive impairment, rapid heart rate, and high blood pressure [4]. Misuse of cathinones has been associated with negative effects including aggressive behavior, self-harm, manic episodes, paranoia, and delusions of persecution; furthermore, cases of acute intoxication can lead to mortality [5, 6]. The abuse of these substances poses many dangers. It is crucial to recognize the potentially detrimental effects of NPSs to address and prevent the associated risks. The 2020 Report on the Drug Situation in China released by the China National Narcotic Control Committee revealed the complex nature of NPSs, which often masquerade as foods and beverages. It is therefore difficult to effectively prevent NPSs use due to its high level of camouflage, allure, and concealment. College students, particularly those transitioning to a new environment and integrating into society, could be more vulnerable to NPSs abuse because of their lack of social experiences [7]. The 2022 World Drug Report revealed a high prevalence of NPSs abuse among individuals aged 16 to 24, particularly in educational settings [2]. When examining the factors influencing knowledge, attitudes, and the propensity to abuse NPSs among university students in China, researchers have reported a significant lack of awareness and understanding regarding NPSs [7], highlighting the urgent need for educational initiatives and interventions to enhance students’ knowledge and awareness of the risks associated with NPSs abuse. To date, two significant models have commonly been used to study substance abuse: the knowledge-attitude-practice (KAP) model [8, 9] and the capability, opportunity, motivation-behavior (COM-B) model [10, 11]. KAP theory is commonly used to explain how individual knowledge and beliefs influence changes in health behavior. It emphasizes the complex interplay between knowledge and behavior and highlights the critical role of attitude change in promoting behavior modification [12]. This theory is particularly well suited for initiatives that aim to raise public awareness and alter attitudes. The COM-B model comprises four key elements: capability, opportunity, motivation, and behavior. It provides a deeper understanding of the psychological mechanisms underlying behavior change and is appropriate for developing behavior intervention strategies and offering targeted recommendations on how to achieve desired behavioral outcomes [13]. The COM-B model is more suited for understanding and intervening in complex behavioral systems, whereas the KAP model is more focused on describing the relationships between knowledge, attitudes, and practices, and assessing how these factors are distributed within specific populations. To enhance the ability of college students to resist NPSs, the KAP model is more appropriate for analyzing the challenges and issues encountered by college students in the context of NPSs abuse prevention education. However, few KAP-based questionnaires have focused on the addictive properties and detrimental effects of NPSs among university students. Currently, although numerous studies have explored the prevalence, demographics, and motivations for NPSs abuse [14–17], the situation in China different from Western countries. In China, NPSs abuse predominantly involve narcotics and psychotropic drugs, and a notable discrepancy exists regarding the perception of drugs across different age groups. Furthermore, Chinese society displays a lower level of tolerance regarding drug-related issues [18]. Therefore, existing research tools may be insufficient for effectively analyzing the challenges faced by Chinese college students in NPSs abuse prevention education. To gain a deeper understanding of college students’ perceptions and behaviors toward NPSs, we need to develop a questionnaire based on the KAP framework that better aligns with our research objective. Additionally, we evaluated the reliability and validity of the questionnaire to ensure its effectiveness and accuracy. The questionnaire developed herein can serve as a useful tool for evidence-based anti-drug educational interventions and behavioral strategies targeted at university students. Methods The initial items were developed based on the KAP framework, a search of domestic and international literature, elements of China’s drug management laws and regulations, and strategies for controlling NPSs [2, 5, 19, 20]. The initial development of the KAP-based questionnaire, item screening, reliability analyses and validity analyses are shown in Fig. 1. Flow diagram of the process of questionnaire validation Phase 1: Questionnaire Development Item generation Four dimensions of NPSs use were identified in this study: demographic dimension about the participants’ demographic characteristics; the knowledge dimension, which encompassed fundamental understanding of NPSs, types of NPSs, associated harms and addictions; the attitude dimension, which encompassed perceptions, emotions, and behaviors toward NPSs; and the practice dimension, which encompassed internal and external behavioral manifestations related to NPSs. The research team engaged in discussions with experts to develop the preliminary questionnaire. Expert consultation The initial version of the KAP-based questionnaire was carefully developed by a multidisciplinary panel comprising experts in drug detection and testing, psychologists, sociologists, and anthropologists. The items underwent a rigorous and comprehensive review process, which involved synthesizing the perspectives of the experts and facilitating deliberations among the panel members. This meticulous approach ensured that each item was thoroughly examined, assessed and refined to meet the highest standards of quality and relevance. Pretest To adapt the questionnaire for university students, a pretest was carried out with a sample of 30 students from schools in the Fujian region. The purpose of this pretest was to assess the variation in their responses, the clarity of the questions, and the absence of ambiguity. Using the results, the main questions were finalized. Questionnaire content Ultimately, the final questionnaire was designed to evaluate four distinct dimensions. (1) The demographic dimension included 10 questions about the participants’ demographic characteristics. (2) The knowledge dimensions included 10 questions, with multiple choice formats for the 4th and 8th items. The response options for these questions followed were “true”, “false”, or “don’t know”. Correct answers received a score of 1, while an incorrect answer or a response of “don’t know” received a score of 0. (3) The attitude dimension included 10 questions that were answered using a five-point Likert scale. These 10 questions comprised 9 negative topics (questions A1-A9) and 1 positive topic (question A10). (4) The practice dimension included 8 questions that were answered using a five-point Likert scale. These items aimed to evaluate the intensity of action, and all of the items were about negative topics (questions P1–P8). For negative topics, the responses ranged from 1 (strongly agree) to 5 (strongly disagree), and the positive topic responses ranged from 1 (strongly disagree) to 5 (strongly agree). Questionnaire deployment This study was carried out in Fujian Province from July to August 2023 and predominantly focused on college students or graduate students at 6 universities. The survey was developed using Questionnaire Star, an online questionnaire software created by Changsha Ranxing Information Technology Co., Ltd. We generated a link to the survey along with a corresponding QR code for data collection via WeChat. Participants completed the survey by scanning the QR code through WeChat. To ensure data quality and completeness, we implemented measures such as limiting each IP address to a single submission and requiring responses to all questions. The responses were then exported to an Excel spreadsheet from the Questionnaire Star platform. Phase 2: statistical analysis To meet project analysis requirements, the number of respondents exceeded 5 to 10 times the number of items. Statistical significance was indicated by p < 0.05. IBM SPSS 29.0 and M-plus 8.0 were utilized for the statistical analyses. Item analysis To screen the items, correlation analysis, Cronbach’s α coefficient, and the critical ratio were utilized. First, the critical ratio method involved selecting participants from the top 27% as the “high-score” group and the bottom 27% as the “low-score” group. At test was conducted to examine the differences between these subgroups. Second, Pearson’s correlation analysis was carried out within each dimension, and correlation coefficients (r) were calculated between item scores and the total score of their respective dimension. Finally, Cronbach’s α coefficient was calculated to evaluate item uniformity, with items removed from the questionnaire if their removal led to an increase in the overall Cronbach’α coefficient. Exploratory factor analysis (EFA) The suitability of the data for factor analysis was determined through Bartlett’s test of sphericity and the Kaiser‒Meyer‒Olkin (KMO) test [21, 22]. A KMO index above 0.6 and a significant Bartlett’s test of sphericity (p < 0.05) indicated the appropriateness of the data for factor analysis. Principal component analysis with varimax rotation was conducted to extract common factors based on eigenvalues greater than 1. Items were considered for deletion if their loading factor was less than 0.4 or if there were two or more loading factors equal to or greater than 0.4 for the same item [23]. After removing items, a new round of analyses was performed to determine the final number of factors. Confirmatory factor analysis (CFA) CFA was conducted to assess the relationship between each item and its designated factor, as well as the relationships between different factors within the construct. The fit of various CFA models was evaluated by examining the change in the χ2 statistic for models using the χ2 test. Additionally, the following parameters were considered to indicate a good fit [24]: root mean square error approximation (RMSEA) < 0.08; comparative fit index (CFI) ≥ 0.90; Tucker‒Lewis index (TLI) ≥ 0.90; and standardized root mean square residual (SRMR) < 0.1. Convergent and discriminant validity Convergent validity was assessed using average variance extracted (AVE) and composite reliability (CR). The discriminant validity was evaluated through the square root of the AVE and the Pearson correlation coefficient. Reliability analysis Reliability was assessed using both Cronbach’s α coefficient and split-half reliability. Split-half reliabilities were calculated by dividing the scores into two sections based on odd and even item numbers. The Pearson correlation coefficient (r) was used to assess the correlation between the two sets of scores, and the Spearman-Brown coefficient was calculated as follows: (). Additionally, Cronbach’s α coefficient was calculated for each of the two parts. Results Item analysis A total of 587 college students participated in this study, of whom 280 (47.7%) were male, 508 (86.5%) were over 18 years of age, 334 (56.9%) were enrolled in a college, 407 (69.3%) were single, 268 (45.7%) had a monthly family income of less than 8000 yuan, and 550 (93.7%) had received anti-drug education (Supplementary Table 1). Items were evaluated for potential removal based on the following criteria: no significant difference in the critical ratio test (t < 4), no significant correlation coefficient (r < 3), and an increase in Cronbach’s α coefficient if the item was removed [25]. The results of these analyses (Table 1, Line 483–484) indicated that item K4_5 had a t value of 0.345 (p = 0.491) and r = -0.010 (p = 0.804), item K4_8 had a t value of 0.661 (p = 0.186) and r = 0.059 (p < 0.150), and item A10 had a t value of 2.388 (p < 0.001) and r = -0.107 (p = 0.009). Moreover, the removal of any of these items led to an increase in the overall Cronbach α coefficient. Table 1Item analysis of the KAP-based questionnaireItemst r Cronbach’s αConclusion Knowledge K1: Most new psychoactive substances (NPS) are modified versions of controlled chemicals (drugs) created by illicit individuals to evade detection, resulting in substances with effects similar or more potent than regulated drugs.10.668**0.654**0.935KeepK2: NPS includes not only chemically altered substances but also natural plant-based materials.9.359**0.608**0.935KeepK3: China has implemented “pre-control” measures on certain NPSs, considering them as the third generation of drugs following traditional drugs and synthetic drugs.13.008**0.702**0.935KeepK4_1: NPS includes: Nitrous Oxide (Laughing Gas)10.993**0.666**0.934KeepK4_2: Ketamine-related substances (Bath Salts, Weed Killer, Jelly)11.659**0.650**0.934KeepK4_3: Ketamine (Special K, Coca-Cola, Fairy Water)13.786**0.730**0.934KeepK4_4: Synthetic Cannabinoids (Bouncing Candy, Little Branches, Natasha, E-cigarette Oil)13.403**0.716**0.934KeepK4_5:3,4-Methylenedioxymethamphetamine (MDMA)0.345 (p = 0.491)-0.010 (p = 0.804) 0.936 Deleted K4_6: Synthetic Opioids (e.g., Fentanyl)11.725**0.706**0.934KeepK4_7: New Benzodiazepine-like drugs (e.g., Zolpidem, Dichloroisopropyl)11.947**0.738**0.934KeepK4_8: NPSs including cocaine0.661(p = 0.186)0.059 (p = 0.150) 0.936 Deleted K4_9: Phenethylamine (Psychedelic Stamps)14.879**0.769**0.934KeepK4_10: Tryptamine (Number Zero Capsules, Firefox, G-spot Liquid)16.676**0.766**0.934KeepK4_11: Fentanyl analogs11.514**0.716**0.934KeepK4_12: Plant-based substances (Chat Grass, Kratom Leaves, Salvia)12.486**0.706**0.934KeepK4_13: Modafinil, Pivaloylacetate (Smart Drugs)13.613**0.765**0.934KeepK5: NPS shares similar physicochemical properties, mechanisms, and effects with drugs.11.469**0.734**0.934KeepK6: Abusing NPS can lead to addiction, withdrawal reactions, harm to personal health, and societal risks, exhibiting both addictive and toxic characteristics.13.085**0.785**0.934KeepK7: New psychoactive substances are often called “milk tea,” “coffee,” “rainbow cigarettes,” and so on. They are packaged to look appealing and can be spread through social media.14.26**0.797**0.934KeepK8_1: Abusing NPS may cause: Accelerated heart rate, increased blood pressure.14.638**0.839**0.934KeepK8_2: Impaired judgment14.638**0.832**0.934KeepK8_3: Blurred consciousness and mental disorders15.387**0.844**0.934KeepK8_4: Induction of accidents, self-harm, and violent behavior14.831**0.825**0.934KeepK8_5: Seizures, coma, shock, or even death14.831**0.851**0.934KeepK8_6: Increased risk of cardiovascular diseases15.8**0.847**0.934KeepK8_7: Elevated cancer risk16.369**0.798**0.934KeepK8_8: Acute poisoning15.23**0.815**0.934KeepK8_9: Liver and kidney function failure15.23**0.840**0.934KeepK8_10: Mania, paranoia, and persecutory delusions14.447**0.842**0.934KeepK9: Inhaling NPS may assist in overcoming opioid addiction.11.848**0.339**0.934KeepK10: NPS use may increase the risk of HIV infection.11.047**0.618**0.935KeepK11: Long-term use of NPS can lead to sexual dysfunction.11.904**0.701**0.934Keep Attitude A1: I think it is acceptable for today’s young people to try NPS once.17.059**0.925**0.932KeepA2: I believe I can control the frequency and dosage of NPS use.16.05**0.878**0.932KeepA3: I think using NPS can enhance the atmosphere of parties or increase popularity at gatherings.17.495**0.953**0.931KeepA4: I believe using NPS can boost my confidence in social situations.17.984**0.962**0.931KeepA5: I think occasional use of NPS can enhance sexual performance.16.843**0.954**0.931KeepA6: I believe using a small amount of NPS can help with weight loss.16.459**0.944**0.931KeepA7: I think using NPS will make me feel energetic.17.209**0.951**0.931KeepA8: I believe using NPS for entertainment and leisure purposes is acceptable.16.576**0.950**0.931KeepA9: I believe that even if I become addicted or dependent on a certain NPS, with effort and abstinence for a period, I can return to the state before using NPS.15.136**0.894**0.932KeepA10: I believe NPSs are drugs 2.388** -0.107(p = 0.009) 0.943 Deleted Practice P1_1: I can easily access NPS in __________. Entertainment venues7.325**0.694**0.934KeepP1_2: Online8.353**0.738**0.934KeepP1_3: Among friends/relatives8.035**0.827**0.933KeepP2: I have used NPS at social gatherings with friends7.951**0.908**0.932KeepP3: I have used NPS when alone7.558**0.922**0.932KeepP4: I have used NPS during sexual activities7.567**0.937**0.932KeepP5: I have used a specific NPS to enhance my attention and memory for important exams7.547**0.920**0.932KeepP6: I have used a certain NPS for weight loss7.497**0.923**0.932KeepP7: I have used a specific NPS to escape reality or reduce life stress7.423**0.918**0.932KeepP8: I have used a specific NPS to break free from limited thinking and enhance creativity7.575**0.907**0.932KeepRemark: **p < 0.001 Item analysis of the KAP-based questionnaire Remark: **p < 0.001 Exploratory factor analysis (EFA) The KMO test and Bartlett’s test of sphericity confirmed the suitability of the remaining items for EFA (KMO = 0.954, χ2 = 37811, df = 1176, p < 0.001). By applying the criterion of eigenvalues greater than 1, a total of 6 factors were obtained through principal component analysis with maximum variance rotation. The rotated component matrix revealed that items K5, K6, K7, K11, P1_2, and P1_3 exhibited multiple loading factors above 0.4, while the loading factor for item K9 was below 0.4. Therefore, these items were considered for elimination (Table 2, Line 485–486). Table 2Rotated component matrix for itemsItemComponent123456K10.655K20.700K30.671K4_10.706K4_20.699K4_30.720K4_40.674K4_60.742K4_70.726K4_90.726K4_100.763K4_110.787K4_120.750K4_130.784 K5 0.498 0.477 K6 0.622 0.457 K7 0.625 0.454 K8_10.853K8_20.826K8_30.860K8_40.823K8_50.870K8_60.854K8_70.805K8_80.810K8_90.865K8_100.834 K9 K100.599 K11 0.427 0.612 A10.895A20.858A30.931A40.930A50.923A60.923A70.933A80.915A90.883P1_10.826 P1_2 0.402 0.794 P1_3 0.608 0.585 P20.869P30.869P40.919P50.909P60.894P70.902P80.895 Rotated component matrix for items The EFA of the remaining 42 items yielded five subdimensions, which were named based on the study design as follows: Fundamental Knowledge of NPSs (Knowledge 1), NPS Types (Knowledge 2), Harms and Addictions Associated with NPSs (Knowledge 3), Attitudes, and Practice. Details of the five subdimensions (items and loading factors) are presented in Table 3. Table 3Structure matrix and item loading factorsKnowledge 1Knowledge 2Knowledge 3AttitudePracticeitemloadingitemloadingitemloadingitemloadingitemloadingK10.733K4_10.714K8_10.854A10.896P1_10.476K20.771K4_20.699K8_20.830A20.859P20.878K30.693K4_30.737K8_30.861A30.930P30.906K100.516K4_40.684K8_40.828A40.931P40.925K4_60.746K8_50.871A50.924P50.915K4_70.737K8_60.856A60.922P60.903K4_90.747K8_70.808A70.931P70.910K4_100.775K8_80.814A80.915P80.898K4_110.791K8_90.869A90.879K4_120.756K8_100.838K4_130.793 Structure matrix and item loading factors Confirmatory factor analysis (CFA) The questionnaire included 42 items, and CFA was conducted to assess a five-sub dimensional model of NPSs-related addiction and harm among university students. The results indicated that the model had a satisfactory fit to the data, with the model fit indices (χ2/df = 3.4, CFI = 0.942, TLI = 0.938, RMSEA = 0.064, SRMR = 0.035) indicating an acceptable fit (Table 4). Table 4Model fit indices of the five-subdimension modelχ2 /dfCFITLIRMSEASRMRAcceptable values< 5> 0.9> 0.9< 0.10.05Test value3.40.9420.9380.0640.035Remark: df = Degree of freedom; CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; RMSEA = Root Mean Squared Error Approximation; SRMR = Standardized Root Mean Square Residual Model fit indices of the five-subdimension model Remark: df = Degree of freedom; CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; RMSEA = Root Mean Squared Error Approximation; SRMR = Standardized Root Mean Square Residual Convergent validity The analysis of convergence validity for the five subdimensions showed that the CR values ranged from 0.78 to 0.99, and the AVE values ranged from 0.47 to 0.88 (Table 5). Table 5Convergence validity analysis of measured items across the five subdimensionsSubdimensionsCRAVEKnowledge 10.780.47Knowledge 20.950.62Knowledge 30.980.84Attitude0.990.88Practice0.970.82Remark: CR = Composite Reliability; AVE = Average Variance Extracted Convergence validity analysis of measured items across the five subdimensions Remark: CR = Composite Reliability; AVE = Average Variance Extracted Discriminant Validity To assess discriminant validity, we used Fornell and Larcker’s verification method [26]. The results revealed that in the knowledge 1 subdimension, the square root of the AVE (0.686) was lower than the maximum correlation coefficient (0.874), indicating a lack of significant discriminant validity. However, the square root of the AVE exceeded the maximum correlation coefficient for the Knowledge 2, Knowledge 3, Attitude, and Practice subdimensions (Table 6). Table 6Results of the correlation analysis between the subdimensions and the square root of the AVEKnowledge 1Knowledge 2Knowledge 3AttitudePracticeKnowledge 1 0.686 Knowledge 20.177 0.786 Knowledge 30.8740.216 0.915 Attitude0.3870.4720.170 0.938 Practice0.4370.3850.1930.392 0.903 Remark: The value of the diagonal is the square root of the AVE Results of the correlation analysis between the subdimensions and the square root of the AVE Remark: The value of the diagonal is the square root of the AVE Reliability The questionnaire exhibited strong internal consistency, as indicated by the Cronbach α coefficients for the overall questionnaire (0.941) and the knowledge, attitude, and practice dimensions (0.968, 0.985, and 0.963, respectively; Table 7). Additionally, the split-half reliability analysis showed that the Cronbach α coefficient was 0.867 for the odd-numbered items and 0.894 for the even-numbered items. The Spearman-Brown coefficient was calculated to be 0.982 (Table 7). Table 7Internal consistency reliability and Split-half reliability of the KAP-based questionnaireoverall questionnaireDimensionssplit-half reliabilityKnowledgeAttitudePracticeoddevenCronbach’s α0.9410.9680.9850.9630.8670.894Spearman-Brown(R)0.982 Internal consistency reliability and Split-half reliability of the KAP-based questionnaire Spearman-Brown (R) Discussion The increasing prevalence of NPSs abuse among university students poses a significant challenge for anti-drug professionals in terms of implementing effective prevention strategies [2, 27, 28]. To gain a better understanding of this issue, it was important to conduct a survey on the addictive and harmful effects of NPSs among university students. The questionnaire developed herein was based on the KAP framework and evaluated various topics related to NPSs, including definitions, laws, types, addictive potential, and associated harms. Researchers commonly utilized methods such as the critical ratio, correlation analysis, and Cronbach’s α coefficient to assess and screen questionnaire items in academic research [29]. The critical ratio’s t value is used to determine item effectiveness, where a t value greater than 4 and a significance level below 0.05 indicate the item’s ability to capture the participant’s response magnitude. Similarly, correlation analysis between items and their respective dimensions is crucial in questionnaire development. A correlation coefficient (r) below 0.3 or a significance level less than 0.05 are commonly used as criteria for item deletion due to inadequate homogeneity with the corresponding dimension. Additionally, an increase in the Cronbach α coefficient when an item is removed indicates that item’s lack of homogeneity with other items [30]. In our study, items K4_5 (“NPSs including 3,4-Methylenedioxymethamphetamine (MDMA)”), K4_8 (“NPSs including cocaine”), and A10 (“I believe NPSs are drugs”) exhibited low t values, nonsignificant correlation coefficients, and homogeneity. Consequently, in accordance with the established criteria, these items were excluded from the analysis because they were unsuitable for assessing university students’ knowledge, attitudes, and practices toward NPSs-related addiction and harm. A subsequent examination revealed that the manner in which these items are presented may potentially result in students making erroneous selections. EFA plays a crucial role in optimizing questionnaires by capturing their internal structure, grouping items with similar theoretical underpinnings, and testing their structural validity [31–33]. In line with previous research, this study utilized factor analysis to categorize items and exclude those that compromised the questionnaire’s structural validity. The analysis revealed that 7 items (items K5, K6, K7, K9, K11, P1_2, and P1_3) had a significant impact on the questionnaire’s structure and were subsequently removed. The remaining items underwent further verification, thus confirming their alignment with the knowledge, attitude, and practice dimensions. Interestingly, within the knowledge dimension, three distinct subdimensions emerged, with items in each subdimension sharing a similar theoretical foundation. The first subdimension focused on the interpretation and definition of NPSs; the second subdimension focused on the classification of NPSs; and the third subdimension focused on the addiction and harm associated with NPSs. This distribution of knowledge dimensions aligns with their underlying theoretical basis. The structural validity of the questionnaire was examined by conducting CFA on the items across the five subdimensions. The fit indices for confirmatory factor analysis include the CFI, TLI, RMSEA, and SRMR [34, 35]. A questionnaire is considered to exhibit a good fit when both the CFI and TLI exceed 0.9, the RMSEA is lower than 0.1, and the SRMR is below 0.05 [25, 36]. The findings of this study demonstrate that the model fit indices for the five subdimensions fall within the optimal range, indicating the validity of the structural model employed in the questionnaire. For convergence validity, an AVE value greater than 0.5 and a CR value greater than 0.7 indicate the strong aggregation validity of items within a dimension [37, 38]. Regarding discriminant validity, if the square root of the AVE for a particular dimension is greater than the correlation coefficient between that dimension and other variables, it signifies a certain level of uniqueness among the items in those dimensions [39]. The Knowledge 2, Knowledge 3, Attitude, and Practice subdimensions of this questionnaire exhibited a high level of aggregation and differentiation. However, it is worth noting that the Knowledge 1 subdimension had an AVE value of 0.47, and its square root was lower than the highest correlation coefficient with other variables. These data suggest that the degree of aggregation and differentiation degree of Knowledge 1 are not optimal. Upon observation, it is evident that the theoretical foundation of item K10 in Knowledge 1 (“The utilization of NPSs may heighten the likelihood of HIV infection”) primarily rests on addiction and harmfulness. Consequently, the inclusion of this subdimension moderately impacts both the aggregation and differentiation levels, thereby justifying its placement within Knowledge 3. Reliability analysis revealed that the Cronbach α coefficients for the entire questionnaire and all dimensions exceeded 0.9, indicating a high level of internal consistency and repeatability. Additionally, the split-half reliability was also acceptable, further confirming the reliability and robustness of the questionnaire. However, there are certain limitations in the compilation of this KAP-based questionnaire. First, the KAP framework is a standard model based on the assumption that perceptions, knowledge, and individual motivations alone are sufficient to induce or instigate behavioral change. However, individuals’ behavior and motivation could be affected by different situations that lead to inaccurate results; therefore, it is important for us to ensure the validity of the questionnaire. Second, the participants were primarily from one city and may not fully represent the entire population of college students; thus, the findings may not be generalizable. Conducting a multicenter survey would be beneficial for validating these findings and ensuring broader representation. Third, considering the small sample size of this study, it is necessary to conduct a large cross-sectional study with follow-up assessments to further explore the efficacy of the KAP-based questionnaire. Conclusion In this study, we developed a valid and reliable KAP-based questionnaire, the final version of the questionnaire consisted of 23 questions with about 42 items, including 6 questions related to knowledge (with multiple choice formats for the 4th and 8th items), 9 questions related to attitudes and 8 questions related to practices, which can serve as a valuable and effective tool for examining university students’ knowledge, attitudes, and practices related to NPSs. Electronic supplementary material Below is the link to the electronic supplementary material. Supplementary Material 1 Acknowledgements The authors wish to extend their appreciation to the anti-drug officers for their administrative support and to the college students in Fujian province who aided and took part in our research. Abbreviations Knowledge, Attitudes, and Practices Novel Psychoactive Substances Exploratory Factor Analysis Confirmatory Factor Analysis Kaiser – Meyer – Olkin Root Mean Squared Error Approximation Comparative Fit Index Tucker-Lewis Index Standardized Root Mean Square Residual Average Variance Extracted Composite Reliability Author contributions Liqin Lu carried out the literature review, designed the questionnaire, collected, analyzed, and interpreted the data, and conceived and wrote the manuscript. Jue Deng collected the data. Fenglan Li participated in data analysis. Xiaoli Zhu supervised the questionnaire design. All authors reviewed and revised the manuscript and counted responsible for its content. Funding This study was support by a grant from Natural Science Foundation of Fujian Province(2024J08116), Fujian Police College 2023 Special Police Project (JW202304) and Fujian Provincial Federation of Social Sciences (FJ2023C04). Data availability The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request. Declarations Ethics approval and consent to participate Ethical approvals were obtained from the Ethics Committee of Fujian Police College (35010410007778). The questionnaire was designed to be anonymous and informed consent was obtained from every respondent. The data were kept confidential, and the results would not identify the respondents personally. Consent for publication Not applicable. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Supplementary Materials Supplementary Material 1 Data Availability Statement The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request.