Among Artificial Intelligence/Machine Learning Methods, Automated Gradient-Boosting Models Accurately Score Intraoral Plaque in Non-Standardized Images Authors: Eric Coy, William Santo, Bonnie Jue, Helen Betts, Francisco Ramos-Gomez, Stuart A Gansky Published: 2024 ABSTRACT AbstractBackground:Previous automated models inaccurately scored non-standardized plaque images. The objectives were to develop and test automated image selection and intraoral plaque-scoring (primary outcome measure in a prevention trial for preschoolers).Methods:Evaluating 1650 plaque-disclosed primary teeth (teeth D, E, F, G) from 435 photographs from UCSF/UCLA clinical trials, data were cleaned, transformed, and modeled with statistical and machine learning (ML) algorithms; data visualizations utilized Jupyter Notebooks, Python, OpenCV, and Sci-kit Learn libraries, with Laplacian filter preprocessing. Image selection and plaque-scoring used 8 ML classification models. Mean plaque-scoring used 8 ML regression models. Models were tuned with 80:20 train:test split, stratified 5-fold cross-validation (5-CV) (unstratified in regression models), and hyperparameter optimization. Area-under-the-curve receiver operating characteristic (AUC-ROC) curve and R2 determined the best classification and regression models, respectively, compared to calibrated dentist researcher ratings. Training time was a secondary metric. Manual segmentation used Photoshop’s lasso tool. Average and dominant hue, saturation, and brightness values were features for training plaque-scoring algorithms.Results:Best performing models were: Support Vector Machine-Gaussian for image selection, 5-CV AUC-ROC of 0.99 and 0.76s of training time; Gradient-Boosting classification and regression models for individual teeth (5-CV AUC-ROC of 0.99 with 105s training); and mean plaque-scoring algorithms (5-CV R2 of 0.72 with 1415s training).Conclusions:Accurate automated plaque-scoring is attainable without the high computational and financial costs of deep learning (DL) models. Automated plaque-scoring is attainable with little user-manipulation.Practical Implications:Implementing automated tooth segmentation and synthetic sample generation with DL training may strengthen reliability, validity, and efficiency for clinical, research, and teledentistry applications by eliminating manual image preprocessing.Keywords: Artificial intelligence/classification, automated pattern recognition/classification, dental photography/classification, dental plaque index/diagnostic imaging, machine learning/classification, observer variation, preschool child Introduction For over 50 years, plaque scoring has seen very little change. In 1964, former United States (US) Assistant Secretary of Health and former UCSF Dean of the School of Dentistry Dr John C. Greene along with Dr Vermillion introduced a “Simplified Oral Hygiene Index” (OHI-S) for standardized plaque scoring1 (Figure 1). At the time, no well-defined, systematic method for classifying oral hygiene existed. The ordinally scaled OHI-S was based on the total surface area of a tooth covered by plaque: 0 (no plaque), 1 (plaque covering <⅓ tooth surface), 2 (⅓≤surface<⅔), 3 (≥ ⅔ surface). Simplified oral hygiene index (OHI-S). 0 (no plaque), 1 (plaque covering < ⅓ tooth surface), 2 (⅓ ≤ plaque < ⅔), 3 (≥ ⅔)1. In clinical settings, accurately classifying intraoral plaque may elucidate the effectiveness of oral hygiene practices, products, and treatments which play a vital role in health outcomes.2 Change in plaque score of teeth is often used as a primary outcome measure for studies in behavioral prevention interventions for children.3,4 Scoring intraoral plaque images is subjective, even by calibrated examiners, as well as tedious, financially burdensome, and sensitive to inaccuracies which may result in misdiagnosis or mistreatment5,6; however, there have been attempts to automate the process of objective intraoral plaque and dental caries classifications through utilizing sophisticated computer algorithms.2,5-12 Classical statistical models have trouble accurately classifying intraoral plaque when images are not captured under ideal and consistent settings.6,7 Controlled lighting environment and patient positioning may be unrealistic requirements in a dental clinic, especially when evaluating pediatric patients. The ultimate goal of developing an automated plaque scoring tool is to create a practical, clinically applicable solution that can be seamlessly integrated into routine dental practice. By providing clinicians with a reproducible, objective, and automated method for assessing plaque accumulation, this tool has the potential to enhance patient care with more accurately monitoring oral hygiene and intervention efficacy/effectiveness. Additionally, the use of a tool that functions well without the need for standardized image conditions could facilitate its adoption in various clinical settings, including those with limited resources. Therefore, the aim of this study was to develop a reproducible, objective, and automated intraoral plaque scoring methodology using shallow machine learning models that do not require standardized image-collection conditions. A secondary, exploratory aim was to determine which image features were more important for automated plaque scoring. Materials and Methods Study Design, Inclusion/Exclusion Criteria, and Data Collection This study was conducted as part of a UCSF/UCLA pilot trial in 2016 and the subsequent Behavioral Economics for Oral Health Innovation (BEECON) randomized trial (ClinicalTrials.gov registration: NCT03862443 and NCT03576326). The study aimed to evaluate the feasibility and acceptability of an incentive-based oral hygiene program. Ethical approval was obtained from the Institutional Review Board (IRB) at UCSF (IRB protocol numbers 16–19920, 16–19968, and 17–23786). Participants included in this study were children aged 6–42 months from two Early Head Start program sites in the Los Angeles area. In total, 36 participants were selected for the pilot trial, and 244 baseline participants were included in the BEECON trial. All participants had at least two fully erupted maxillary incisors. A total of 435 intraoral images were collected, depicting 1650 plaque-disclosed primary maxillary incisors across three visits for the pilot trial and one baseline visit for the BEECON trial. To visualize plaque, a study-trained staff member applied Butler GUM Red-Cote Dental Disclosing Solution (Sunstar Americas, Inc., Schaumburg, IL) during in-person visits. All participating parents provided informed consent including consent to photograph their child’s teeth. In the pilot trial, both a Photomed Canon G16 camera with ring flash (PhotoMed International, Van Nuys, CA) and an iPhone7 (Apple, Cupertino, CA) were used for photography. Based on pilot results, only the iPhone7 was utilized in the main trial due to its higher image quality and ease of evaluation.3 Plaque scoring was performed by an experienced and calibrated dentist researcher using the Simplified Debris Index (DI-S) of the Simplified Oral Hygiene Index (OHI-S), modified for only facial surfaces of maxillary incisors (OHI-MIS).1 This individual then scored plaque and recorded other data points, including plaque disclosing solution thickness (categorical), score confidence (categorical), and image clarity (blurry/clear) (Figures 2a, b and 4). (a) Sample UCSF/UCLA Pilot trial plaque photograph classified as “clear”. Primary maxillary incisors were disclosed with GUM red-cote Liquid. (b) Sample UCSF/UCLA Pilot trial plaque photograph the program classified as “blurry”. BEECON Pilot and baseline mean plaque score (OHI-MIS). There was class imbalance in the dataset with plaque scores averaging 2.2. The reliability of the rater was high, with Lin’s concordance correlation coefficient >0.77 inter-rater and >0.83 intrarater.13-15 Image Preprocessing Environment and Tools All operations were performed within a Jupyter Notebook environment in the Python 3.7 programming language.16 Operations included data cleaning and transformations, statistical modeling, data visualizations, and the development, training, and testing of machine learning processes. There were no minimum requirements for input data, as all images were of the same data format (JPG) and no images had missing accompanying data points designated by the calibrated dentist researchers. The computer used to create and run the programs utilized a 1.6 GHz Dual-Core Intel Core i5 processor, 8 GB 2133 MHz LDDR3 memory, and an Intel UHD Graphics 617 1536 MB graphics card. Image Quality Filtering Images were preprocessed with a Laplacian filter without customized parameters.17 A 3 × 3 kernel of values specific to the Laplacian filter was passed through each pixel, and then each pixel (denoted as the “index pixel”) was compared to all the pixels surrounding it. If the values of the surrounding pixels were vastly different than the index pixel, then this index pixel is likely to be part of a contour within an image that is sharp, not blurry. A script written using the OpenCV library performed this comparison for every pixel within an image.17 This library contains specialized pre-written scripts and algorithms that allow a developer to execute complex functions for computer vision operations without the need to write them entirely from scratch. The more pixel values within an image considered as in the range of sharp versus blurry, the more likely the whole image is sharp. As a machine learning model cannot feasibly use every pixel in an image to determine whether it is blurry or not, the Laplacian values’ mode and standard deviation from each image were utilized for classification. Tooth Segmentation and Color Correction After filtering out the poor quality images, teeth were individually isolated using Photoshop’s lasso tool.18 This allowed precise tracing and segmenting of all plaque-disclosed primary maxillary incisors from their respective images. Portable network graphic (PNG) transparency remained in the image files, so a script was constructed to ignore the alpha channel in each image. This script consisted of using a binary mask to differentiate between the transparency (alpha) channel and RGB (red, green, blue) channels. Photoshop also allowed color-correction of 11 images by displacing the hue channel by +86 units. This was necessary to repair images suffering from improperly set configurations on the Canon camera used to take those 11 otherwise clear photographs. Feature Extraction After segmentation, hue, saturation, and value (brightness) (HSV) components were extracted from each pixel of the segmented teeth using the OpenCV library.17 These were the image features (variables) used for the automated scoring process in which plaque score was the target (outcome) variable. HSV components of all pixels in each image were subsequently grouped into dominant H, S, and V components and average H, S, and V components for dimension reduction. Classical Statistical and Machine Learning Hyperparameters Eight machine learning and classical statistical models were fitted to the data to determine which images were too blurry and could not be used for reliable plaque classification analyses (Figure 3). The eight models used were as follows: support vector machine with Gaussian kernel (SVM-G), SVM with a linear kernel (SVM-L), Gaussian naïve Bayes, linear discriminant analysis (LDA), logistic regression, gradient boosting, K-nearest neighbor, and random forest. Project workflow. Images and their features were systematically processed and classified. The same eight models were used for developing the single-tooth classification tool. The regression models used for the 4-teeth mean plaque score tool were Epsilon support vector regression with a radial basis function (SVR-rbf), ridge regression, gradient boosting regression, adaptive boosting (AdaBoost) regression, random forest regression, stacking regression, and bootstrap aggregation (bagging) regression. Hyperparameters for the gradient boosting classification and regression models were as follows: learning_rate = 0.005, n_estimators = 750, max_depth = 4, min_samples_split = 8, min_samples_leaf = 1, max_features = 7, subsample = 1. Average and dominant HSV data from each photo were exported into Sci-Kit Learn library’s model-fitting function, which fitted accompanying models with variables to estimate the plaque score target.18 One program was developed to predict the plaque score for each segmented tooth; a second program was developed to predict the mean plaque score for all the segmented teeth within an image (teeth D, E, F, G in the US (universal) numbering system or equivalent teeth 52, 51, 61, 62 in the Federation Dentaire Internationale (FDI) numbering system). Both have valuable uses: single-tooth scoring may allow the ability to uncover the precision at which a patient or hygiene product is or is not cleaning a specific surface of interest; the average score of four teeth may give a more generalized sense of patient oral hygiene effect. Models were tuned with a train-test-validate split to combat overfitting, stratified 5-fold cross-validation (unstratified in regression models) to allow samples of each class into each fold, and hyperparameter optimization to find the right combination of values that are best suited to build the model architectures. Eighty percent of the data were used for training, 10% for testing, and 10% for validation. Stratification allowed even distribution of plaque scores in training, test, and validation sets, thus, ensuring plaque scores were not disproportionately misrepresented into folds. 5-fold cross-validation is appropriate so that there were enough samples in the sets to evaluate models while balancing the bias-variance tradeoff.19 Data Evaluation The predictive performance of the models was assessed by calculating the area-under-the-curve receiver operating characteristic (AUC-ROC) curve and R2 for 5-fold cross-validated data. Program run-time was also collected to determine whether training classical statistical and machine learning models for intraoral plaque scoring can be done in a subjectively reasonable amount of time. In a research setting, long training times may deter consideration of using a specific model. The best model was chosen by selecting the one with the highest validation AUC-ROC/R2 and ties broken with shortest run-time. Results Image Quality Filter All models trained, tested, and validated were objectively accurate, with the highest accuracy provided by the SVM-G with an AUC-ROC of 0.99 and the lowest accuracy provided by LDA with an AUC-ROC of 0.83 (Figure 5). Thirty-one of 435 images were determined by a SVM-G to be of too poor quality to be scored (Figure 6). These photos were not included to develop the automated plaque scoring program. The program could still attempt a plaque score determination with these images; however, it would not be useful to the user as there is no way to validate the accuracy of this output without accompanying scores from a calibrated dentist researcher. Image quality program results. Of all models, SVM-G provided the highest accuracy. Laplacian pixel values. A clear delineation can be observed between most blurry and sharp image Laplacian values. Individual Plaque Score Classification and Mean Plaque Score Regression Of all classification and regression models, gradient boosting classification and regression models were able to accurately score images of individual teeth and 4-teeth (#D-G) with AUC-ROC and R2 of 0.99 and 0.72, respectively (Figures 7-8). Single tooth plaque classification results. Of all models, gradient boost and SVM-G provided the highest validation accuracy. 4-teeth mean plaque score results. Of all models, gradient boost provided the highest validation accuracy. Over all possible thresholds, a plaque score of 0 was correctly classified most frequently by the gradient boosting classification model, whereas a plaque score of 2 was incorrectly classified most frequently by the gradient boosting classification model (Figure 9). On a fixed threshold, a plaque score of 3 was correctly classified most frequently by the gradient boosting classification model, whereas a plaque score of 0 was incorrectly classified most frequently by the gradient boosting classification model (Figure 10). Individual plaque score classification AUC-ROC. Class imbalance may lead to inaccurately scoring classes with fewer observations. Normalized confusion matrix. Plaque score 3 was correctly scored the most frequently. Training time varied greatly among the models. The more parameters used, the longer the run-time. For example, the gradient boosting classifier has 21 parameters and took 105s of training-time to train, whereas logistic regression has 15 parameters and took 0.18s of training-time (Figure 7). Still, these run-times are quite fast (i.e., negligible) in the realm of machine learning. A feature importance program allowed evaluating which pixel values contributed to more accurate scoring of plaque (Figure 11). Dominant saturation contributed the most to accurate plaque scoring. Feature importance. The dominant saturation (dom S) pixel value of a segmented tooth image contributed most to the development of an accurate plaque scoring model. Discussion This study successfully addressed the primary research question of developing a reliable (reproducible), objective, and automated intraoral plaque scoring methodology using shallow machine learning models without the need for standardized image acquisition conditions. The models demonstrated high accuracy in classifying plaque scores, as evidenced by the high AUC-ROC values across different model architectures. These results suggest that the proposed methodology can effectively overcome the challenges posed by variations in image quality and collection conditions, which are often encountered in real-world clinical settings. Models were chosen based on the public accessibility of script functions provided in the Sci-Kit Learn library.16 Models were initially chosen using Sci-Kit Learn’s flowchart “Choosing the right estimator”. The flowchart indicated, and statistician coauthors confirmed that classification and regression models were appropriate for our project in which classification is used to predict categorical data (plaque scores 0–3) and regression is used to predict numerical data (mean plaque scores). Although the maximum margin is very narrow between the positive and negative hyperplanes of the support vector machine (SVM) that delineates sharp and blurry classifications of our images, this model fits these data well. There is a clear separation between Laplacian values for the two classes. The non-linear characteristic of a Gaussian kernel, while seemingly unapparent in our figure, may also account for subjectivity in classifying an image as blurry since there are some images that have a field of view that creates a sharp focal point and a blurry background. One could argue that a SVM with linear kernel would be also appropriate since its AUC-ROC was 0.99 too but required a longer (but still reasonable) run-time of nearly one second. Due to the gradient boosting model’s built-in ability to construct successive training sets from incorrectly classified predictions, this approach is favorable for classification and regression problems with imbalanced classes.20 This explains why gradient boosting models performed best for single-tooth classification and 4-teeth mean regression. The class imbalance is clear since there were very few images containing teeth with plaque scores less than 2 (Figure 4). Models had little opportunity to learn what scores of 1 and 0 “look like.” However, this is not reflected in the AUC-ROC of the single-tooth classification model in which plaque scores of 0 and 1 appear to be more accurately classified than plaque scores of 2. This may be because so few plaque scores of 0 and 1 were misclassified. Average-Precision or Precision-Recall Curves may be better metrics for comparing the performance of classification models in the presence of class imbalance as in our dataset.21 Dominant saturation having the most influence on the model’s determination of an accurate plaque score makes sense. Abstract as it may be to consider, this feature may primarily be what a clinician considers when visually inspecting the disclosing solution on teeth. The more plaque there is on a tooth, the more disclosing solution adheres to it. In turn, the solution appears more concentrated, which translates into a more saturated color and a higher plaque score. Conversely, the less plaque there is on a tooth, the less disclosing solution adheres to it. In turn, the solution appears less concentrated, which translates into a less saturated color and a lower plaque score. One limitation of the study is the relatively small size of the dataset, which consisted of 435 intraoral images of 1650 teeth. Another limitation of the study was not considering additional image features in the analysis. For example, incorporating shape and size information about the teeth or plaque, as well as information about the surrounding oral cavity, could have improved the performance of the models. Additionally, with a more balanced dataset, the models could be better trained. Oversampling data from underrepresented low plaque classes (i.e., 0 and 1) would help balance the data, but this could lead to overfitting. Alternatively, collecting more data is another way this problem can be remedied. However, machine learning models tend to reach a plateau in their predictive power as the amount of training data increases, whereas performance scales with increasing sample size while training with deep learning (DL) models.22-27 An important limitation of this study is that the plaque scoring data used to train the machine learning models was generated by only two calibrated dentist researchers. While both examiners were highly calibrated and showed strong inter-rater and intra-rater reliability (Lin’s concordance correlation coefficient >0.77 inter-rater and >0.83 intra-rater), having input from additional observers could have further strengthened the quality of the data and increased its generalizability. Ideally, including plaque scores from at least three or more observers would minimize the potential for bias introduced by individual examiners and provide a more robust dataset for training. Furthermore, multiple examiners could help capture a broader spectrum of variability in scoring techniques, making the automated model even more reliable when applied across diverse clinical settings. In future studies, it may be beneficial to include a larger panel of calibrated observers to ensure that the input data better represents the range of clinical variability present in real-world plaque scoring. Despite the relatively small and imbalanced dataset, the models performed well, indicating that the approach is robust even under suboptimal conditions. However, the limitations highlighted in this study – particularly the small sample size and class imbalance – suggest areas for future improvement. Addressing these issues, such as through the collection of more data or the use of oversampling techniques, could further enhance the performance of the models. Additionally, the exploration of DL models, which can automatically learn features from raw data, offers a promising avenue for future research that could extend the capabilities of the current methodology. DL differs from machine learning in that we do not need to select the features (variables) for the program to use, but rather the program decides itself what to look for as it discovers what works to improve its accuracy and what does not.28 A future direction in this project will be to use DL to strengthen our models when UCSF IT’s DL resources are operational. Unfortunately, DL requires a lot of processing power because of the immense number of parameters that are used in calculations – for example, the AlexNet convolutional neural network (CNN) has 62,378,344 parameters (versus 21 parameters for gradient boosting) and, therefore, can be expensive to implement when considering labor-hours, fees for product licensing, and equipment run-time.29 Two models from DL that would help here are: a generative adversarial neural network (GNN), which synthesizes new data, thus reducing the need to collect more data,30 and a CNN, which will allow us to automatically segment the teeth from the image and identify the plaque without thresholding HSV components.31 A CNN would also take into consideration the presence of varying lighting conditions, patient positioning, and extraneous objects. Conclusion From a clinical perspective, implementing an automated plaque scoring tool holds great promise for improving patient care. By providing clinicians with an easy-to-use, objective method for evaluating oral hygiene, this tool can enhance the accuracy of plaque assessments and support more informed decision-making. This, in turn, can lead to more personalized and effective treatment plans, better patient adherence to oral hygiene practices, and ultimately, improved oral health outcomes. As this technology continues to develop, its integration into clinical workflows could become standard practice, benefiting both clinicians and patients alike. These plaque scoring tools could potentially be integrated into an electronic health record (EHR) system or a teledentistry application, eliminating the need for subjective and time-consuming plaque scoring. This may lead to less time for the patient in the chair, better health status documentation and fewer scoring errors, and more efficient clinician time for other procedures. On a research level, projects that require large numbers of disclosed teeth to be scored could be completed in a shorter amount of time with excellent reliability. Accurate automated plaque-scoring is attainable without high computational and financial costs of training deep learning (DL) models. Automated plaque-scoring is attainable with little user manipulation; however, this tool still requires manual segmentation. Implementing automated tooth segmentation and synthetic sample generation with DL training may strengthen validity for clinical, research, and teledentistry applications by eliminating manual image preprocessing. In conclusion, this study successfully developed a reproducible, objective, and automated plaque scoring methodology using shallow machine learning models that do not require standardized image acquisition conditions. The high accuracy of the models in classifying plaque scores underscores the potential of this approach to be implemented in clinical settings, offering a practical tool for improving patient outcomes. Future work should focus on expanding the dataset, addressing class imbalance, and exploring the integration of deep learning techniques to further refine and enhance the methodology. Acknowledgments Thank you to the John C. Greene Society for the privilege to be a part of the research fellowship program, Dr Stuart Gansky for his mentorship, Bill Santo for his teachings, Dr William Jacobson for the OHI-S illustration, and the UCSF School of Dentistry for the opportunity of a lifetime. Requests for access to the code developed for these programs should be made to the corresponding author. There are no licenses to reuse the code, and there is no restriction for access. Funding Eric Coy’s predoctoral research fellowship was supported by the Delta Dental Community Care Foundation; research reported in this publication was supported by the United States National Institute of Dental & Craniofacial Research of the National Institutes of Health under Award Numbers UH2/UH3DE025514 with support from the Coordinating Center under Award Number U01DE025507. This work is also supported in part by the United States National Institute of Dental & Craniofacial Research of the National Institutes of Health under Award Number U01DE033240. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Additional in-kind support was provided by Philips Healthcare Sonicare for Kids and Colgate-Palmolive. Footnotes Disclosure Statement No potential conflict of interest was reported by the authors. References