Detection of risk levels in optical mark recognition exams using categorical feature-based machine learning models Authors: Yasin Sancar Published: 2025 ABSTRACT Today, the evaluation of large-scale examinations is mostly carried out using optical marking systems. Although these systems provide fast and accurate scoring, they often overlook behavioral patterns on answer sheets. Features such as blank responses, double markings, repetitive patterns, and response diversity can support a more comprehensive assessment of exam security and student behavior. This study aims to predict students’ risk levels from optical marking data using machine learning. The target variable consists of expert-defined five-level risk labels. Since all features were categorical, OrdinalEncoder and 5-fold stratified cross-validation were applied. Among all models, CatBoost achieved the highest performance on the five-level scale (test accuracy 73%, weighted F1 0.72, macro-F1 0.69) and clearly outperformed a rule-based baseline constructed from expert thresholds (13% accuracy, macro-F1 0.11). Confusion matrix and ROC analyses showed high discriminative power, with minor confusion between adjacent classes. Additionally, following a consolidated three-level setting, the model achieved 91% accuracy and 0.85 macro-F1, improving stability for decision-making. The results show that optical marking data reveal meaningful behavioral patterns and enable reliable risk assessment beyond simple correct-incorrect analysis. Introduction In recent years, measurement and evaluation processes in education have undergone a significant transformation due to the impact of digitalization. Optical mark recognition (OMR) systems are widely used, especially in the evaluation of multiple-choice exams administered in open and distance learning systems. While these systems offer fast, low-cost, and objective scoring, they also have some weaknesses in terms of data reliability and exam security1. The use of OMR technologies in education allows for the analysis of not only right-wrong scoring but also students’ behavioral tendencies during exams2. The literature has identified various behavioral risks that threaten exam security. These risks include excessive blanking, double marking, consecutive identical answers, low marking diversity, and block or symmetrical marking patterns. Such patterns may indicate student inattention, lack of motivation, a tendency to answer randomly, or attempts to cheat3,4. However, manually examining such behaviors, especially in large-scale exams, is a time-consuming process prone to human error. Therefore, the development of automatic detection systems has become inevitable to ensure exam security in education. In recent years, machine learning-based methods have come to the fore in exam security and automatic analysis of student behavior. Decision trees, support vector machines, ensemble algorithms, and deep learning approaches are used to detect cheating attempts or abnormal behavior with high accuracy from students’ exam response patterns5,6. In this context, systems developed for risk detection in educational data make significant contributions to enhancing exam security. In the educational context, the analysis of student behavior is being researched not only for security purposes but also for predicting learning outcomes and performance assessment. For example, machine learning-based models have been developed to predict learning outcomes using student behavior data in online or hybrid learning environments7. Similarly, the impact of different exam formats and assessment practices on students’ achievement levels has also been examined8. These studies show that student behavior provides valuable information not only in terms of exam security but also in terms of improving the quality of education. However, the vast majority of existing studies address student exam behavior using a binary classification (normal/abnormal) approach and do not provide a risk level analysis graded according to the severity of the risks. However, when low-risk anomalies (e.g., a few blank questions) and high-risk anomalies (e.g., systematic block patterns or excessive double marking) are evaluated in the same category, sufficient detail is not provided for educational administrators’ decision-making processes. Therefore, multi-level risk classification offers a more functional solution in the context of exam security. In this study, a five-level risk classification model was developed using student response patterns labeled by expert opinions on optical marking data: “Very Low,” “Low,” “Medium,” “High,” and “Very High.” The models obtained using machine learning algorithms (Logistic Regression, Naive Bayes, KNN, Decision Tree, Random Forest, Gradient Boosting, AdaBoost, XGBoost, LightGBM, and CatBoost) were evaluated using 5-fold cross-validation, and the algorithms with the best performance were analyzed in detail on the test data using metrics such as confusion matrix, classification report, ROC, and Precision-Recall curves. The contributions of the study can be summarized as follows: 1. Multi-level risk classification was performed, going beyond the binary classification approach commonly found in the literature. 2. An explainable and interpretable decision support model was developed by combining expert labeling with machine learning methods. 3. The proposed approach was designed as a practical and automated solution that can be integrated into large-scale exam applications. In conclusion, this study aims to enhance exam security, objectively analyze student behavior, and improve educational quality. It has been demonstrated that OMR data is a valuable resource not only for scoring but also for behavioral analysis. Thus, it contributes to more comprehensive and data-driven decision-making processes for education administrators. Literature review The study, which provides a comprehensive assessment of the technical aspects and practical limitations of OMR, details issues such as mark reading errors, paper/form quality, and obtaining reliable data with low-cost hardware9. The use of OMR in the context of continuous assessment has been reported in terms of process efficiency and regular monitoring of student performance in short tests and assignments10. On the hardware/algorithm side, a low-cost and robust OMR pipeline has been proposed, presenting design principles aimed at improving reading accuracy in real-world conditions1. This line of work implies that OMR data is not limited to “correct-incorrect” scoring; behavioral signals (blanking, double marking, patterns) can also be captured. Kamalov et al. presented a machine learning-based approach for detecting cheating in exams; they examined the predictive power of behavioral indicators by comparing multiple classifiers3. Meng discussed the concept of profiling in cheating detection and proposed a framework for how behavior-based feature sets (session-level interactions, response times, etc.) can be modeled in an ethical and explainable manner11. Findings focusing on assessment practices in education emphasize that online exam design and the tools used significantly influence student behavior8. This group of studies demonstrates that meaningful detection success can be achieved with relatively low-cost data sources (log/response data). Zhen and Zhu proposed a TabNet-based ensemble approach, presenting a framework that combines classical ML with deep representations in tabular data and reported strong results in plagiarism detection12. Alsabhan attempted to capture anomalies from students’ interaction sequences over time using LSTM-based sequential models13. Erdem and Karabatak designed a system for detecting cheating in online exam environments using a deep learning + classical ML hybrid; they demonstrated the benefit of jointly handling feature extraction and final classification layers4. This approach offers advantages, particularly in capturing the time/sequence dimension and hybrid feature interactions. Liu et al. addressed cheating detection and localization together using a MIL approach, demonstrating that it is possible to produce decisions at the segment level (time window/segment) from weakly labeled data (e.g., “suspicious/clean” at the session level)14. These results are important for answering the question “where did the event occur?” especially in dense data sources such as video/logs; although not directly transferable to single-modal systems such as OMR, they are methodologically instructive for weakly supervised modeling. A system that recognizes cheating behavior in exam rooms using deep learning from camera data has been proposed; clues such as facial expressions, gaze, and body orientation have been utilized15. Similarly, multidimensional cheating detection in paperless exam scenarios has been reported using image processing and multi-perspective matching16. A visual intelligence system based on body posture and facial features has been developed for classroom environments, focusing on behavior classification accuracy and real-time monitoring17. These studies demonstrate that high detection performance is possible with multimodal data, but limitations such as ethics/privacy and infrastructure costs hinder scalability. Systems that predict grades/success based on student behavior characteristics emphasize that behavioral indicators (interaction intensity, duration, diversity) are strong predictors7. In blended courses, output prediction was performed using online learning behavior data; regular relationships between interaction patterns and success were reported18. While this line of work is not directly focused on cheating detection, it clarifies the relationship between behavioral feature engineering and output modeling and contributes to exam security research in terms of data architecture. Systematic reviews detail the strengths and weaknesses of supervised/unsupervised methods, the imbalanced class problem, and the critical role of metric selection (especially PR-AUC, macro-F1)5. Network/cybersecurity-focused reviews and method studies highlight topics such as hybrid approaches, feature selection, cost-sensitive optimization, and calibration6,19,20. These principles offer directly applicable practical recommendations for rare/high-risk classes in educational contexts (e.g., class weighting, resampling, confidence score reporting). In classical assessment literature, the impact of multiple-choice tests on learning and their design quality have been discussed; the relationship between item writing, distractor quality, and test strategies with student outcomes has been examined2,21. Comparative reviews of assessment approaches in distance education highlight consistency and security requirements in the evaluation practices of online programs22–24. This framework supports the pedagogical legitimacy and administrative feasibility of plagiarism/anomaly detection efforts. The Cheating Detection Triangular Model, developed for sensitive areas such as surveys, offers a modeling approach for structurally detecting behavioral inconsistencies25. Although not applied directly to exam paper data, it reinforces the idea of pattern-based inconsistency and feeds the logical basis for rule-based pre-screening (e.g., excessive blanking, sequential repetition, low entropy). Generally, most of the literature relies on a binary (normal/abnormal) detection paradigm; graded (multi-class) risk modeling is relatively limited. Although multi-modal and high-accuracy approaches are becoming more prevalent, there remains a need for single-modal, low-cost solutions such as OMR on the cost-privacy-scale axis. Explainable frameworks that combine behavioral/rule-based pre-screening with ensembles suitable for categorical data3,11,12 are candidates to fill this gap; particularly ordinal risk modeling, cost-sensitive thresholds, and uncertainty/calibration reporting are recommended future directions for administrative use and ethical transparency5,6. Methods Dataset In this study, a large-scale dataset obtained from multiple-choice exams conducted within the scope of open and distance education was used. The dataset contains over 529,131 student-exam session records. Each record represents a student’s entire answer sheet for a specific course exam and contains responses to 20 multiple-choice questions. The questions are formatted with five options (A-E). Unmarked answers are coded as “N” while double-marked (invalid) answers are coded as “*.” This structure enables not only correct/incorrect analysis but also the examination of behavioral patterns exhibited by students during the exam. For example, a very high number of blank answers or long consecutive sequences of the same option provide meaningful indicators in terms of exam security. The rule of “4 wrong answers cancel out 1 correct answer” in exams leads students to leave answers blank rather than mark them randomly. In exams where the minimum passing grade is 35, approximately 7 correct answers are sufficient, enabling students to strategically leave up to 13 questions blank. In this context, leaving 14 or more questions blank is considered risky in terms of exam security. Table 1 shows the explanation of behavioral features the dataset. Feature extraction Raw optical marking data has been converted into various behavioral and statistical features for risk-oriented behavioral screening. In this process, both exam regulations and metrics proposed in previous studies have been taken into account. The main features obtained are as follows: number of questions left blank, number of double markings, number of different options used, number of repetitions of the most frequently chosen answer, length of the longest consecutive sequence of the same answer, entropy of the answer distribution, detection of cyclic patterns, detection of symmetric patterns, and block marking behavior. These features statistically represent the systematic or random answering strategies exhibited by the student during the exam. The entropy metric was computed to quantify the diversity of responses. It is defined as:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H(x) = - \sum _{i=1}^{k} p_i \log p_i \end{aligned}$$\end{document}H(x)=-∑i=1kpilogpiwhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_i$$\end{document}pi denotes the normalized frequency of each response option and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k=5$$\end{document}k=5 is the total number of answer choices (A–E).Table 1Behavioral features extracted from optical mark recognition data.FeatureDescriptionnum_blankThe number of questions left blank by the student. High values may indicate low effort, time constraints, or deliberate blanking strategies on the exam.num_doubleNumber of questions marked more than once. Indicates carelessness, indecision, or uncertainty in exam strategy.longest_consecutive_sameThe longest series of consecutive identical answers. Monotonous marking may indicate low motivation or a tendency to give random answers.num_unique_answersThe number of different answer options used by the student. Low diversity may indicate a monotonous response strategy.entropyThe entropy value of the response distribution. A low value indicates uniformity, while a high value indicates excessive diversity.cycle_patternA binary variable that indicates whether there is a specific cyclical pattern in the responses.symmetric_patternA binary variable that indicates whether responses exhibit a symmetrical structure.block_patternA binary variable indicating the concentration of responses in specific blocks on the same option. Table 1Behavioral features extracted from optical mark recognition data.FeatureDescriptionnum_blankThe number of questions left blank by the student. High values may indicate low effort, time constraints, or deliberate blanking strategies on the exam.num_doubleNumber of questions marked more than once. Indicates carelessness, indecision, or uncertainty in exam strategy.longest_consecutive_sameThe longest series of consecutive identical answers. Monotonous marking may indicate low motivation or a tendency to give random answers.num_unique_answersThe number of different answer options used by the student. Low diversity may indicate a monotonous response strategy.entropyThe entropy value of the response distribution. A low value indicates uniformity, while a high value indicates excessive diversity.cycle_patternA binary variable that indicates whether there is a specific cyclical pattern in the responses.symmetric_patternA binary variable that indicates whether responses exhibit a symmetrical structure.block_patternA binary variable indicating the concentration of responses in specific blocks on the same option. Behavioral features extracted from optical mark recognition data. Rule-based pre-filtering After the attributes were removed, rule-based pre-filtering was performed to identify potential behaviors that could compromise exam security. The rules used at this stage were developed based on pedagogical logic, exam regulations, and measurement and evaluation principles. As a result of applying these rules, 10,377 of the 529,131 records were flagged as potential risks for violating at least one rule. As seen in Table 2, the applied feature thresholds form the basis of the risk detection rules.Table 2Criteria and thresholds of behavioral features to flag potentially risky marking behaviors.FeatureCriteriaNumber Of RecordDescriptionnum_blank\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 14$$\end{document}≥141674Leaving a large number of questions blank may indicate low effort during the exam, lack of time, or a strategic decision not to answer. A score of 14 or higher means that more than 70% of the exam was left blank.num_double\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 2$$\end{document}≥21553Two or more double markings may indicate carelessness, indecision, or a technical problem with the answer sheet. Since a single double marking may sometimes be accidental, the threshold value has been set at 2.longest_consecutive_same\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 6$$\end{document}≥6927Marking the same answer 6 or more times in a row increases the likelihood of low motivation or guessing, especially in tests.num_unique_answers\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\le 3$$\end{document}≤34944Using only 3 or fewer different answer options indicates uniform marking or deliberate strategic restraint behavior.entropy\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\le 0.5$$\end{document}≤0.5256Low entropy, which measures the diversity of responses, indicates that the answers are extremely uniform.entropy\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 2.5$$\end{document}≥2.50Although high entropy theoretically indicates excessive diversity and randomness, no such situation was observed in the data set.cycle_pattern== 1697The presence of a specific cyclical pattern in the answers may indicate that the exam strategy has been manipulated outside of its natural flow..symmetric_pattern== 1298Symmetrical response patterns may indicate deliberate copying, deception, or attempts to draw attention.block_pattern== 12885Answers concentrated in blocks on a particular option are indicative of exam strategy or deliberate marking behavior. Table 2Criteria and thresholds of behavioral features to flag potentially risky marking behaviors.FeatureCriteriaNumber Of RecordDescriptionnum_blank\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 14$$\end{document}≥141674Leaving a large number of questions blank may indicate low effort during the exam, lack of time, or a strategic decision not to answer. A score of 14 or higher means that more than 70% of the exam was left blank.num_double\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 2$$\end{document}≥21553Two or more double markings may indicate carelessness, indecision, or a technical problem with the answer sheet. Since a single double marking may sometimes be accidental, the threshold value has been set at 2.longest_consecutive_same\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 6$$\end{document}≥6927Marking the same answer 6 or more times in a row increases the likelihood of low motivation or guessing, especially in tests.num_unique_answers\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\le 3$$\end{document}≤34944Using only 3 or fewer different answer options indicates uniform marking or deliberate strategic restraint behavior.entropy\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\le 0.5$$\end{document}≤0.5256Low entropy, which measures the diversity of responses, indicates that the answers are extremely uniform.entropy\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 2.5$$\end{document}≥2.50Although high entropy theoretically indicates excessive diversity and randomness, no such situation was observed in the data set.cycle_pattern== 1697The presence of a specific cyclical pattern in the answers may indicate that the exam strategy has been manipulated outside of its natural flow..symmetric_pattern== 1298Symmetrical response patterns may indicate deliberate copying, deception, or attempts to draw attention.block_pattern== 12885Answers concentrated in blocks on a particular option are indicative of exam strategy or deliberate marking behavior. Criteria and thresholds of behavioral features to flag potentially risky marking behaviors. Figure 1 shows the percentage equivalents of the same matrix. This representation allows for a relative interpretation of the frequency of co-occurrence. For example, 25.7% of students who meet the num_blank \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\>= 14$$\end{document}=14 rule also meet the num_unique_answers \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 3$$\end{document}≥3 rule. Similarly, 58.7% of students with the block_pattern == 1 rule also comply with the longest_consecutive_same \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 6$$\end{document}≥6 rule. This co-occurrence analysis shows that risk rules do not appear independently but are mostly clustered. In particular, the high frequency of rules involving blanking (num_blank \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge 14$$\end{document}≥14), limited number of different answers (num_unique_answers \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\le 3$$\end{document}≤3), and block patterns (block_pattern == 1) indicates the presence of distinct risk patterns in the exam papers.Furthermore, these rules were also used to construct a rule-based baseline classifier for comparison in the evaluation stage. The rule-based filter can be expressed formally as:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R(x)= {\left\{ \begin{array}{ll} 1 & \text {if } (num\_blank \ge 14) \vee (num\_double \ge 2) \vee (longest\_consecutive\_same \ge 6) \vee \\ & (num\_unique\_answers \le 3) \vee (entropy \le 0.5) \vee \\ & (cycle\_pattern = 1) \vee (symmetric\_pattern = 1) \vee (block\_pattern = 1) \\ 0 & \text {otherwise} \end{array}\right. } \end{aligned}$$\end{document}R(x)=1if(num_blank≥14)∨(num_double≥2)∨(longest_consecutive_same≥6)∨(num_unique_answers≤3)∨(entropy≤0.5)∨(cycle_pattern=1)∨(symmetric_pattern=1)∨(block_pattern=1)0otherwisewhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R(x)=1$$\end{document}R(x)=1 denotes a record that violates at least one behavioral threshold.Fig. 1Co-occurrence matrix of risk detection rules (%). Co-occurrence matrix of risk detection rules (%). Expert labeling In this study, the final classes were determined by blind evaluation of records that passed rule-based pre-screening by domain experts. The process consists of the following steps: (i) preparation and creation of a codebook, (ii) pilot labeling and calibration, (iii) main labeling (double-blind), (iv) consensus and arbitration, (v) reliability analysis, and (vi) quality assurance. Preparation and codebook. Before labeling began, operational definitions based on behavioral indicators were developed for the five-level risk scale (“Very Low,” “Low,” “Medium,” “High,” “Very High”). Expected patterns for each class (number of blanks, length of consecutive identical options, response diversity, entropy, block/cycle/symmetry signals) and contextual interpretations of these indicators (e.g., the effect of the “4 wrongs cancel 1 right” rule on blank labeling decisions; distinguishing between strategic blanking and exam abandonment) were detailed. The codebook includes “example-counterexample” sections, gray area decision trees, and notes on how to handle erroneous forms (e.g., double marking). Pilot and calibration. Experts (at least two evaluators) independently labeled a pilot package consisting of 100–200 records. A calibration session was then held to address cases with differing opinions; threshold and priority rules were clarified in the codebook (e.g., the principle that “a single long sequential sequence is not high risk on its own if other indicators are normal, but at most moderate risk”; escalation in the presence of multiple long sequences). Main labeling (double-blind). Records were anonymized, with the student_id and course_id fields masked. Each record contained only a 20-question answer sequence (A-E, “N” “*”) and attribute summaries (num_blank, num_double, num_unique_answers, longest_consecutive_same, entropy, cycle/symmetric/block flags) and small visualizations (answer histogram, conditional sequence diagram) were displayed. Experts selected a single class per record; if necessary, they added a short free-text note. The labeling interface recorded the timestamp and version information of decisions, leaving an auditable trail (audit trail). Conciliation and arbitration. When two experts’ labels differed, the system initiated a conciliation session: first, consensus was sought through joint review and codebook reference; if no agreement was reached, a third arbitrator made the final decision. The arbitrator only reviewed cases of disagreement; blindness was maintained. Reliability analysis. Inter-observer agreement for the double-blind section was calculated using Cohen’s K, and the metrics (K, percentage of agreement per class) were reported in the Findings section. The agreement level is in the “high” range; however, the accumulation of examples in gray areas at class boundaries is an expected finding and is discussed from a pedagogical perspective in the Discussion section. Quality assurance and balance. The distribution of labels was monitored at regular intervals, and codebook principles were maintained to prevent specific rules from becoming overly decisive on their own (e.g., a high blank rate alone should not be moved to “Very High” without context; accompanying patterns should be sought). When serial labeling (block) and symmetric sequences were detected, the multi-indicator principle was observed. Additionally, labeled data was used in a manner respectful of the training/test split to prevent overfitting in machine learning; if a student had multiple sessions, data leakage was prevented by ensuring they did not fall into the same partition. The method used in the study is an integrated structure that combines rule-based filtering, expert labeling, and machine learning processes. Optical marking data is first filtered according to behavioral rules, then labeled by experts according to risk levels, and finally, machine learning models are trained on this labeled data. This approach ensures that the data is processed in a systematic and controllable manner from its raw state to the decision support stage. Figure 2 shows the general structure of the data processing and modeling flow applied in the study.Fig. 2End-to-end pipeline of the proposed method from raw OMR data to risk prediction and decision support. End-to-end pipeline of the proposed method from raw OMR data to risk prediction and decision support. Modelling In this study, the target variable is risk_level_expert, and the features consist only of students’ responses (q1-q20). The identity variables were removed during preprocessing to prevent data leakage. Moreover, all variables were categorical; therefore, OrdinalEncoder was used in the preprocessing, and unknown_value=−1 was assigned for categories not seen during training to preserve generalization. To prevent data leakage, encoding was defined in ColumnTransformer and executed together with the classifier via Pipeline. Missing values and unexpected symbols were standardized and converted to a single format prior to encoding. The data was stratified into an 80% training and 20% testing ratio, thus preserving class proportions in both subsets. Preliminary checks showed that the class distribution was balanced; therefore, SMOTE/SMOTENC, class_weight, or any other resampling technique was not applied. Stratified 5-Fold Cross-Validation was used on the training set to reliably estimate generalization performance, and all models were evaluated with the same stratum splits (shuffle, random_state=42). The algorithms compared are: Decision Tree, Random Forest, Gradient Boosting, XGBoost, LightGBM, CatBoost, Logistic Regression (multi-class), Linear SVM, Gaussian Naive Bayes, K-Nearest Neighbors, and AdaBoost. The applications were implemented using the scikit-learn, xgboost, lightgbm, and catboost libraries. The evaluation metrics are Accuracy, Precision, Recall, Macro-F1, and Weighted-F1, which are calculated on a class-by-class basis in the multi-class case and then averaged using macro/weighted averages. ROC/PR analyses were performed using the One-vs-Rest approach, and micro and macro averages were obtained. After the final model selection, the selected model was retrained on the entire training data and reported on an independent test set (results are presented in the Results section). For all randomness sources, random_state=42 was fixed, and the experiments were made reproducible. For the multi-class case, the evaluation metrics were defined as:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} F1_{macro} = \frac{1}{K}\sum _{i=1}^{K} F1_i \end{aligned}$$\end{document}F1macro=1K∑i=1KF1i4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} F1_{weighted} = \sum _{i=1}^{K} w_i \cdot F1_i \end{aligned}$$\end{document}F1weighted=∑i=1Kwi·F1i5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Accuracy = \frac{TP+TN}{TP+FP+TN+FN} \end{aligned}$$\end{document}Accuracy=TP+TNTP+FP+TN+FN Results In this study, a 5-fold stratified cross-validation method was applied to evaluate class prediction performance. This method made it possible to measure the generalizability of models while preserving class distribution in each stratum. As shown in Table 3, the CatBoost model achieved the highest results among all models, with an average accuracy of 0.7118, a weighted F1 score of 0.6989, and a macro average F1 score of 0.6701. This success is related to CatBoost’s category-aware handling of data and regularization strategies that reduce overfitting. LightGBM (0.6905 accuracy, 0.6854 weighted F1) and XGBoost (0.6906 accuracy, 0.6831 weighted F1) showed similar performance, but fell behind CatBoost in macro-F1 due to weaker predictions on minority classes. Simpler models such as Logistic Regression (0.4627 accuracy) and KNN (0.5078 accuracy) showed the lowest performance. In particular, linear and distance-based methods were found to be inadequate for multi-class and fully categorical data structures. The three top-performing models in the cross-validation stage (CatBoost, LightGBM, XGBoost) were then evaluated on an independent test set.Table 3Cross-validation performance results of machine learning models.ModelCV_Acc_MeanCV_F1w_MeanCV_F1m_MeanCatBoost0.71180.69890.6701XGBoost0.69060.68310.6511LightGBM0.69050.68540.6544RandomForest0.67930.65490.6065GradBoost0.64400.62030.5710NaiveBayes0.58420.56070.5031DecisionTree0.53760.53860.5088KNN0.50780.48400.4526LogReg0.46270.38530.3191LinearSVC0.46120.37790.3122 Table 3Cross-validation performance results of machine learning models.ModelCV_Acc_MeanCV_F1w_MeanCV_F1m_MeanCatBoost0.71180.69890.6701XGBoost0.69060.68310.6511LightGBM0.69050.68540.6544RandomForest0.67930.65490.6065GradBoost0.64400.62030.5710NaiveBayes0.58420.56070.5031DecisionTree0.53760.53860.5088KNN0.50780.48400.4526LogReg0.46270.38530.3191LinearSVC0.46120.37790.3122 Cross-validation performance results of machine learning models. On the test set, CatBoost again maintained its leading performance, achieving 73% accuracy, 0.72 weighted F1, and 0.69 macro-F1, demonstrating the consistency of cross-validation and test results. This indicates that CatBoost is the most effective model for evaluating category-based behavioral features in the dataset. In addition to machine learning models, a simple rule-based classifier based on expert-defined thresholds listed in Table 4 was also evaluated. This method achieved only 13% accuracy and a macro-F1 score of 0.11. This result demonstrates that expert rules alone are insufficient for prediction and that the CatBoost model does not simply mimic the rules. The machine learning model can learn and generalize more complex decision patterns that go beyond predefined rules.Table 4Performance comparison of the rule-based approach and the CatBoost model.ModelAccuracyWeighted F1Macro F1Rule-based Classifier0.130.100.11CatBoost (5 level)0.730.720.69 Table 4Performance comparison of the rule-based approach and the CatBoost model.ModelAccuracyWeighted F1Macro F1Rule-based Classifier0.130.100.11CatBoost (5 level)0.730.720.69 Performance comparison of the rule-based approach and the CatBoost model. Fig. 3Confusion matrices of the top three models on the test set. Confusion matrices of the top three models on the test set. Figure 3 presents the confusion matrices of the three best performing models on the test set. The CatBoost model provides the most balanced performance across all five classes. In particular, it correctly identifies the Very Low and Low risk levels with high recall (0.89 and 0.78) and high f1-score (0.86 and 0.80), indicating strong predictive capability on the most frequently occurring classes in the dataset. While High and Medium classes also exhibit acceptable accuracy (0.84 and 0.63 recall), the majority of misclassifications occur between these two adjacent levels. The Very High class is the most challenging to predict due to its behavioral similarity to High risk patterns and a lower sample size, resulting in lower recall (0.31). LightGBM and XGBoost show similar trends, particularly strong performance on Very Low and Low levels (recall values around 0.89 and 0.78), but both models suffer from increased confusion in the Medium and Very High classes. The Medium class, in particular, is frequently shifted to either Low or High, indicating overlap in behavioral features around the central risk boundary. XGBoost performs better than LightGBM in the High class (recall 0.77 vs. 0.75), yet still falls short of CatBoost in macro-level stability. Overall, these results reveal that the majority of classification errors occur around the middle risk categories (High, Medium, Very High), suggesting natural behavioral overlap between the boundaries of these classes. CatBoost, however, maintains higher performance consistency across all risk levels and produces the lowest confusion rates among the three models.Fig. 4ROC and Precision-Recall curves of the CatBoost model on the test set. ROC and Precision-Recall curves of the CatBoost model on the test set. Figure 4(a) illustrates the multi-class ROC curves of the CatBoost model. The model demonstrates high discriminative capability across all classes, with micro- and macro-average AUC values of 0.94 and 0.93, respectively. The Very Low and Low classes achieve the highest AUC values (0.97 and 0.95), indicating excellent separability. The Medium and Very High classes show slightly lower but still strong performance (0.93 and 0.91), while the High class obtains the lowest AUC (0.89). These results confirm that the classifier captures the dominant patterns of low and high risk groups, while boundary-class overlaps naturally occur around the middle risk levels. Figure 4(b) presents the Precision-Recall curves. Consistent with the ROC results, the PR curves exhibit high average precision in the Very Low (0.88) and Low (0.83) classes, whereas the High and Very High classes show lower precision values (0.59 and 0.64), reflecting greater behavioral variability and fewer examples. The Medium class again lies in the middle (0.67 AP). Overall, both ROC and PR analyses demonstrate strong predictive performance and confirm the robustness of the proposed model across multiple risk levels. Feature importance analysis In addition to model evaluation metrics, a feature importance analysis was conducted using the CatBoost model to examine the contribution of each behavioral feature to prediction performance. The analysis results indicate that the most decisive variables are, in order, num_blank, block_pattern, and longest_consecutive_same. These features provide a strong discriminative effect for risk classes by capturing strategic blanking behavior, dense marking clusters, and consecutive response patterns, respectively. While other variables such as response diversity and entropy also contribute to the model, the relative impact of symmetric and cyclic pattern variables was observed to be lower. This analysis demonstrates that the model does not adhere to a single rule; rather, it uses multiple behavioral criteria in a complementary manner. Furthermore, this finding strengthens the model’s applicability in educational settings and offers a significant advantage in terms of decision transparency. Three-level consolidation experiment Following the evaluation of the five-level risk classification, it was determined that the class boundary uncertainties observed in the medium-level risk groups affected the model performance. This situation is particularly evident between the High-Medium and Medium-Low classes. In addition to this observation, an alternative labeling experiment was conducted in which the expert label was reduced from five levels to three levels, in line with the referee’s suggestion. The transformation was defined by the following rules;Very Low and Low \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}→ Low RiskMedium \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}→ Medium RiskHigh and Very High \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}→ High RiskThis transformation is based on two main reasons: (i) the similarity of exam behaviors in medium-risk categories, (ii) the fact that risk assessments in real applications mostly require decision-making at three levels. The CatBoost model was retrained with a three-level label space, preserving all data preprocessing steps, cross-validation structure, and testing procedures used in the five-level model. The three-level model achieved 0.91 accuracy, 0.91 Weighted F1, and 0.85 Macro F1 on the test set. This result demonstrates that class consolidation increases model stability and reduces the uncertainty observed in boundary classes. At the same time, the model was able to preserve general behavioral patterns independently of the five-level label and performed well even with a simpler decision structure. Very Low and Low \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}→ Low Risk Very Low and Low \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}→ Low Risk Medium \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}→ Medium Risk Medium \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}→ Medium Risk High and Very High \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}→ High Risk High and Very High \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}→ High Risk Discussion In this study, risk level estimation was performed using expert labels from optical marking data, and the performances of different machine learning algorithms were compared. The findings revealed that tree-based ensemble methods, particularly CatBoost, LightGBM and XGBoost, demonstrated the highest performance in both cross-validation and the test set. CatBoost’s superior performance can be attributed to two main reasons: (i) the model’s ability to process category-based features natively, thereby preventing information loss during the data preprocessing stage, and (ii) the effective use of regularization parameters that reduce overfitting risk and sampling strategies that maintain class balance. In addition to machine learning models, a simple rule-based classifier constructed directly from expert-defined thresholds was also evaluated. This baseline achieved only 13% accuracy and 0.11 macro-F1, demonstrating that the expert rules alone are insufficient for reliable classification and that the CatBoost model does not merely reproduce the rule set. Instead, the machine learning model is capable of learning more complex behavioral patterns that generalize beyond the predefined criteria. LightGBM and XGBoost also achieved relatively high accuracy rates, but they were not as consistent as CatBoost, especially for rare risk levels (e.g., Very High). This difference can be attributed to CatBoost’s ability to better preserve inter-class discrimination through its internal handling of categorical variables and target-encoding-like mechanisms. Confusion matrix analyses showed that misclassifications were more prevalent in medium-risk classes (especially Medium). This can be explained by the fact that the feature distributions of these classes are similar to both low and high risk levels, making it difficult for the models to distinguish between them. Similarly, the relatively lower ROC-AUC and Average Precision values observed in the High and Very High classes, compared to the Very Low and Low classes, may be due to the smaller number of examples and higher behavioral variability in these groups. The literature also frequently emphasizes that imbalanced or rare-class distributions reduce prediction performance in such settings. As an additional experiment, the original five-level labels were consolidated into three broader risk categories, following the reviewer’s recommendation and based on the continuity of behavioral characteristics. The three-level CatBoost model significantly improved performance (91% accuracy, 0.91 weighted F1, 0.85 macro-F1) and reduced the ambiguity observed in boundary classes, indicating that the risk structure in optical marking data may inherently follow a more compact categorical formulation in practical scenarios. This result suggests that, in operational contexts, a three-level scale (low, medium, high risk) may offer a good balance between interpretability and predictive stability. ROC and PR analyses confirm that CatBoost has high discriminative power across all classes, but the AUC value in the High class is lower than in other classes. This situation can be attributed to the overlap in behavioral characteristics and the fact that exam strategies create transitions between certain risk levels. In particular, “neighboring class” errors are an expected finding in situations where exam performance is not sharply separated by thresholds, but behavioral metrics vary continuously across a spectrum. All student data used in this study has been anonymized, and model outputs are not used for any punitive, decision-making, or individual classification purposes for any student. Risk levels serve only as diagnostic and early warning indicators to support exam security and assessment processes. This classification does not aim to label or discriminate against individuals; it merely flags behavioral patterns to facilitate expert review. Therefore, the ethical framework for using model outputs is a supportive process that includes expert verification and institutional evaluation steps. Among the limitations of the study is that the data was obtained from a single exam system and a specific student population. Including different exam types, difficulty levels, and more heterogeneous student profiles could increase the generalizability of the models. Additionally, the three-level consolidation experiment suggests that alternative class formulations can be explored as a future direction for improving model robustness and interpretability. These findings, consistent with similar studies in the literature, demonstrate that category-based ensemble methods are successful in extracting complex patterns from training data. The results obtained reveal that optical marking data not only enable true/false analysis but also allow for an in-depth evaluation of student behavior. Conclusion In this study, the aim was to predict marking risk levels using optical marking data and expert labels. The data was structured using only the students’ responses (q1-q20), and identity variables were excluded during preprocessing to prevent data leakage. Since all features are categorical, OrdinalEncoder was used in the preprocessing process and a 5-fold stratified cross-validation method was applied in the evaluation of the models. The findings show that tree-based ensemble methods, especially CatBoost, achieved the highest accuracy, macro-F1 and weighted-F1 scores in both cross-validation and the independent test set. The CatBoost model achieved 73% accuracy, 0.72 weighted F1 and 0.69 macro-F1 in the test set. A simple rule-based baseline constructed from expert thresholds achieved only 13% accuracy and 0.11 macro-F1, demonstrating that the machine learning model does not merely replicate rule-based decision boundaries. Confusion matrix and ROC/PR analyses revealed that the model achieved high performance in low and high risk levels, while neighboring class errors were concentrated around the Medium and High levels. These patterns reflect the continuity of behavioral characteristics in exam performance. In addition, an alternative three-level formulation improved model stability even further (91% accuracy, 0.91 weighted F1, 0.85 macro-F1), suggesting that a more compact risk structure may be more suitable for practical applications. The results obtained reveal that optical marking data provide behavioral indicators that go beyond correct and incorrect answer evaluation. Such analyses offer the opportunity to interpret student behavior and exam performance in a multidimensional way, and can support both exam security and assessment processes. As future work, new behavioral metrics and explainability methods (e.g., SHAP or LIME) can be integrated, and additional datasets with different exam types and student groups can be evaluated to improve generalizability. These findings demonstrate that artificial intelligence and machine learning techniques can be used reliably and effectively in optical marking-based exam evaluation processes.