Neural network for natural language processing to determine treatment urgency in an ophthalmology emergency department Authors: Anna Hillenmayer, Bjoern Lofi, Sabrina Langhans, Carolin Elhardt, Armin Wolf, Christian Maximilian Wertheimer Published: 2026 ABSTRACT AbstractBackgroundIn an ophthalmology emergency department, determining treatment urgency is crucial for patient safety and the efficient use of resources. The aim of this study was to use artificial intelligence to develop a neural network and evaluate its accuracy in predicting treatment urgency.MethodsIn a retrospective study, a medical history questionnaire consisting of a free-text section and checkbox questions was given to 1715 patients on arrival and the responses were used as input data. The data were labelled as either emergency or non-urgent according to the patients’ diagnosis and divided into a training and a test dataset. A bidirectional encoder representations from transformers (BERT) neural network for language processing was concatenated with a multilayer perceptron (MLP), and the result was processed by another MLP. Performance metrics were determined on the test dataset. Particular emphasis was placed on explainable artificial intelligence methods.ResultsThe combination of BERT and MLP achieved the highest F1 score of 0.81, with sensitivity of 0.89 and specificity of 0.58. The three ophthalmologists achieved F1 scores of 0.77, 0.75 and 0.73, respectively. Most of the input features that were identified by explainable artificial intelligence methods as being responsible for the network’s decision correspond to clinically relevant features for the emergency/non-urgent decision.ConclusionsA neural network was developed to identify treatment urgency based on natural language data and checkbox questions which could complement the initial assessment. Future studies could focus on validation and improving accuracy, as well as integrating the network into workflows and discussing ethical implications. WHAT IS ALREADY KNOWN ON THIS TOPIC In an ophthalmology emergency department, determining the urgency of treatment is critical to patient safety and resource efficiency. Emergency department overcrowding has been identified as a global problem, and delayed patient care has been associated with poorer outcomes. Using artificial intelligence to support medical decision-making has become an important clinical and scientific issue, with prediction accuracy, accountability and ethical considerations playing a major role in developing new neural networks. In an ophthalmology emergency department, determining the urgency of treatment is critical to patient safety and resource efficiency. Emergency department overcrowding has been identified as a global problem, and delayed patient care has been associated with poorer outcomes. Using artificial intelligence to support medical decision-making has become an important clinical and scientific issue, with prediction accuracy, accountability and ethical considerations playing a major role in developing new neural networks. WHAT THIS STUDY ADDS An artificial intelligence model using natural language processing has been developed to analyse text provided by patients for use in supporting decisions in the early stages of patient flow management, and it has demonstrated acceptable accuracy and performance. Most of the input features that influence responsible for the network’s decision, identified by explicable artificial intelligence methods, correspond to features that would be expected to be clinically relevant for emergency/non-emergency. An artificial intelligence model using natural language processing has been developed to analyse text provided by patients for use in supporting decisions in the early stages of patient flow management, and it has demonstrated acceptable accuracy and performance. Most of the input features that influence responsible for the network’s decision, identified by explicable artificial intelligence methods, correspond to features that would be expected to be clinically relevant for emergency/non-emergency. HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY The proposed neural network design demonstrated the potential for artificial intelligence to be a useful addition to the initial assessment process in ophthalmology emergency departments and future studies might focus on potential validation in daily clinical practice, generalisability, further improvement of accuracy, integration into existing workflows and thorough discussion of ethical implications. The proposed neural network design demonstrated the potential for artificial intelligence to be a useful addition to the initial assessment process in ophthalmology emergency departments and future studies might focus on potential validation in daily clinical practice, generalisability, further improvement of accuracy, integration into existing workflows and thorough discussion of ethical implications. Introduction Eye emergency department (ED) crowding has been identified as a global problem, with delayed patient care being linked to poorer outcomes.1 In this context, EDs are often busy, resulting in longer waiting times and potential delays in medical care for critical cases. One study found that non-urgent conditions account for nearly half of all ED visits.2 It is therefore crucial to identify patients whose condition does not warrant emergency care and who could be seen in a more controlled setting within the next few days.3 Typically, trained ophthalmology nurses perform initial assessments to triage patients based on the urgency of their eye conditions. This ensures that those with critical eye conditions receive prompt treatment, while those with less serious issues do not delay treatment for others.4 As there are many factors to consider, systems have been developed to assist nurses and other staff in the decision-making process. These include simple tools such as checklists and scoring systems5 as well as more advanced methods such as machine learning6 and deep learning models.7 Deep learning has recently gained popularity for extracting information from multidimensional and multifactorial inputs.8 In our ED, we currently use a combined questionnaire to determine treatment urgency and medical history, including a freeform written ophthalmological complaint as well as checkbox questions. The questionnaire part with categorical answers can be easily processed with classical deep learning methods. However, the free-form written complaint section requires natural language processing, and using newer deep learning techniques is required to extract the treatment urgency from the text. The bidirectional encoder representations from transformers (BERT) neural network language model is able to process natural language in the form of text or speech and classify text into groups.9 The BERT network can be fine-tuned by training on any natural language recognition task. In this study, we developed a deep learning artificial intelligence approach to determine treatment urgency in an ophthalmology ED using multimodal, multifactorial input. The neural network receives input in the form of a combination of a checkbox style questionnaire and a free-form written ophthalmological complaint. The output is a suggestion for treatment urgency to assist the on-call staff with their decision making. Due to the black box nature of the neural network decision in this critical setting, emphasis is placed on methods for explainable artificial intelligence methods. Methods Study design This single-centre retrospective study included a total of 1715 patients presenting to our fully staffed 24-hour university-based ophthalmic ED between May 2023 and November 2024. Inclusion criteria for the study were the completion of a combined paper-based symptoms and medical history questionnaire with a free-form written complaint box at the time of presentation and if they had a documented diagnosis of the reason for presentation, which was reached at the end of the visit after full evaluation. Patients with more than three missing items or an absent written acute medical complaint were excluded from the study. The study adhered to the tenets of the Declaration of Helsinki and was approved by the local ethics committee of the University of Ulm (approval ID: 388/23). Questionnaire A two-page questionnaire was completed by each patient on admission. The first part consisted of a free-text box for ocular complaints. The second part consisted of checkbox questions regarding time of onset, whether the symptoms were related, red and green flags and the reason for presentation. These two sections were used for processing in the neural network. The third section of the questionnaire, on the second page, is dedicated to the patients’ general medical history and asks for example about general diseases, allergies and cardiovascular risk factors. The questionnaire was digitised using a self-developed digital questionnaire tool with a graphical user interface. All checkbox items were presented to the neural network as ‘yes’, ‘no’ or ‘missing’ in the form of a one-hot coded tensor. Thus, the neural network was also provided with the information, if a missing value is inputted. The text was tokenised for the BERT model. Labelling The diagnosis that led to the ED visit and that was reached by the ophthalmologist after a thorough slit-lamp examination, including visual acuity and tonometry at the end of the ED visit, was obtained from the digital patient record. All diagnoses were grouped into forty diagnostic subgroups. Each subgroup was assigned a label of ‘emergency’ or ‘non-urgent’ by three different ophthalmologists and the final decision was made by consensus or two to one vote. A substantial inter-rater reliability of 0.8 was reached. Neural network design A custom-built neural network was used to classify treatment urgency, processing two different types of natural language data from free-text and checkbox questions as input. The deep learning library PyTorch was used, with the integrated development environment PyCharm V.2023.3.2 and Python V.3.12. All coding was done by CMW, and all neural networks were stored and run on a single workstation with 7 CUDA devices (NVIDIA GeForce RTX 4090). BERT, a large language model, was chosen for its ability to analyse and classify text in context. An open-source version that had been pretrained on a German-language dataset was used for this study. It can be fine-tuned for specific tasks, is open-source and requires less computational power than other language models, enabling efficient self-training on our workstation. The German version of the BERT model was provided by the MDZ Digital Library team at the Bavarian State Library, Munich, Germany. The source code of the pretrained network is publicly available and was trained by fine-tuning using our free-text responses. A rectified linear unit-activated multilayer perceptron () processed the checkbox style responses. Both networks were concatenated and again processed by another hidden linear layer. The output probability was provided as a single integer between 0 and 1, and the cut-off for class identification was set to 0.5. The previously normalised dataset was transformed into batches and tensors using csv, and the Torch library and the MDZ Digital Library Team’s tokeniser were used with the data loader. The neural network was trained on the training dataset, and an increase in the accuracy of this training dataset was used to trigger a checkpoint save of the trained network. A systematic grid search was performed to determine the most useful hyperparameters. Training was stopped if there were more than 20 epochs where the accuracy in the training dataset did not increase (figure 1). The two types of data used in this manuscript were processed using a neural network design tailored for this purpose. The free-text natural language was processed by a pretrained German version of the BERT model with 12 repetitions of the hidden layers, which was fine-tuned on our data. The checkbox questions were processed by a classical multilayer perceptron. The output of both networks was concatenated and again processed by a single linear layer. A single output neuron gave the output probability after processing in a sigmoid layer. Zero was non-urgent and one marked an emergency with a cut-off of 0.5. BERT, bidirectional encoder representations from transformers; ReLU, rectified linear unit. Explainable artificial intelligence To explain the decisions made by the neural network, the Captum model interpretability library which is based on PyTorch was used alongside different attribution algorithms. Attribution with integrated gradients, gradient shap, deepLIFT and ablation allowed to identify the main features of the input data from the checkbox question answers used by the MLP neural network to reach its output. For the German BERT model, LayerIntegratedGradients and interpretation hooks were used to identify the tokens (word parts) from the input text that influenced the network’s decision. Statistical analysis All calculations were performed in Python V.3.12 within the PyCharm IDE V.2023.3.2 using the following libraries: csv, scipy, statistics, pandas, scikit-learn and numpy. The performance of the test dataset, which had not been encountered by the neural network, and the decisions made by three ophthalmologists were evaluated using scikit-learn and seaborn libraries in Python. In the confusion matrix metrics, the label emergency was counted as a test positive or ‘one’. Graphs were plotted using either matplotlib.pyplot V.3.5.3 or GraphPad PRISM V.10 (GraphPad Software, San Diego, California, USA). A t-test was used to compare the different groups and a Pearson χ2 test was performed for categorical comparisons using scipy.stats plugin in Python. A Cochran’s Q-test with a Bonferroni post hoc correction was used to determine if one model outperforms another based on the distributions of correct versus incorrect decisions on the test dataset and Fleiss Kappa for interrater agreement was calculated using SPSS V.30 (IBM, Armonk, New York, USA). A p value <0.05 was considered statistically significant. Results Demographics A total of 1715 patients were divided into two groups based on their final diagnosis at the end of their emergency visit. 63.6% were classified as emergency cases, while 36.4% were classified as non-urgent. The most prevalent diagnoses in the non-urgent group were conjunctivitis (269), hordeolum (80) and dry eye (128), whereas corneal erosion (193), mostly metallic external foreign bodies (219) and entoptic retinal symptoms without retinal tear (105) were the most common diagnoses in the emergency group (figure 2A). The mean age was 43±21 years, while the emergency group was on average 5 years older (p<0.0001). Symptoms had been present for a shorter period of time in the emergency group, and check-box answers indicating sharp or blunt trauma, a foreign body or entoptic symptoms were significantly more common (p<0.001) (figure 2B). Responses indicating eye swelling, current use of eye drops, conjunctivitis and red eye were more prevalent in the non-urgent group (p<0.001). Pain on the pain scale and visual loss on the subjective visual loss scale were statistically equivalent in both groups (p>0.07) (table 1). (A) Eye-related emergency department visits. The demographic data of presenting eye emergencies are shown by diagnosis and label. The actual diagnosis that was reached at the conclusion of the visit was grouped into 40 categories (illustrated as bubbles) and then assigned to one of two triage groups (n=1715; on the graph, all names of diagnostic subgroups with fewer than 40 cases are omitted). Green=non-urgent, red=emergency cases. (B) The questionnaire consisted of items asking directly about specific symptoms, circumstances and reasons for presentation. The categorical occurrence of responses to the given item was statistically analysed. Several items were statistically different between the two groups, suggesting the ability of the questionnaire to differentiate between the groups (n.s.: p>0.05, *p<0.05, ****p<0.0001). Table 1Distribution of ophthalmology department visits into "urgent" or "non-urgent"AllEmergencyNot urgentSignificanceN17151091 (63.6%)624 (36.4%)Age43±2145±2140±22<0.0001Pain (0–11)4.2±2.94.2±3.04.2±2.70.7Subjective vision loss(0–10)3.4±2.93.5±3.03.3±2.80.07The groups were unevenly distributed, with 63.6% of patients being categorised as urgent and 36.4% as non-urgent. The age of the patients was significantly higher in the urgent group (p < 0.0001). Pain severity and subjective visual loss were not statistically different between the groups. The age of the patients was significantly higher in the urgent group (p<0.0001). Pain severity and subjective visual loss were not statistically different between the groups. Distribution of ophthalmology department visits into "urgent" or "non-urgent" The groups were unevenly distributed, with 63.6% of patients being categorised as urgent and 36.4% as non-urgent. The age of the patients was significantly higher in the urgent group (p < 0.0001). Pain severity and subjective visual loss were not statistically different between the groups. The age of the patients was significantly higher in the urgent group (p<0.0001). Pain severity and subjective visual loss were not statistically different between the groups. Neural network performance The performance metrics were determined using a test dataset that was withheld and had not been encountered by the neural network previously. Combining BERT and MLP to process both the free-text eye complaint box and the check box questions numerically achieved the highest F1 score of 0.81. The use of only the check box questions or only the natural language import produced slightly lower but comparable results at 0.8. The ophthalmologists had between five and 11 years of work experience and had to make their decisions using the test dataset reaching F1 scores of 0.77, 0.75 and 0.73, respectively. Even though the F1 scores of the ophthalmologists were comparable, there was only a moderate interrater reliability of 0.5. Integrating BERT with an MLP, a statistically significant higher number of accurate predictions than the ophthalmologists can be observed (p=0.02 for ophthalmologists 1 and 2 and p<0.001 for ophthalmologist 3). Confusion matrix metrics suggested that the neural networks were better at identifying emergencies, resulting in higher recall (sensitivity). However, the ophthalmologists were better at identifying those patients who were not labelled as emergencies, resulting in higher specificity (figure 3A–C). (A) The receiver operating characteristic (ROC) curve suggests equal performance between the different types of input data and the specific neural network used. The performance of the ophthalmologists (OPHTHA) was comparable. (B) Confusion matrices highlight differences between specificity and sensitivity of ophthalmologists compared with neural networks. (C) Overview of performance metrics for different neural network inputs and the three ophthalmologists. In our binary output classification, the performance was calculated using several confusion matrix metrics comparing the predicted labels with the true labels. An emergency was defined as a positive test. BERT, bidirectional encoder representations from transformers; eme, emergency; MLP, multilayer perceptron; n.u., non-urgent. Explainable artificial intelligence Using explicable artificial intelligence methods, we identified most of the input features responsible for the network’s decision. These features correspond to those that would be expected to be clinically relevant for the emergency/non-urgent decision. In the German BERT natural language processing network, our model focuses on words related to the symptoms or the mechanism of injury described by the patient. For instance, ‘emergency’ was predicted by words such as ‘stab’, ‘finger’ and ‘severe’ for corneal erosion or ‘black’ and ‘flashes’ for entoptic symptoms without retinal break. In contrast, non-urgent cases included allergy, conjunctivitis and hordeolum, which were associated with words such as swollen, itchy, red and tearing (figure 4A). Four different attribution algorithms were calculated for the checkbox questions: ablation, integrated gradients, deepLIFT and gradientSHAP. For example, in response to the direct question regarding sharp or blunt trauma, foreign body and entoptic symptoms, selecting a ‘yes’ answer led the network to categorise the case as an emergency, while selecting ‘no’ led to a categorisation as non-urgent. Older age and the absence of lid swelling were associated with emergency. Conversely, red eye, no recent surgery and conjunctivitis led to a non-urgent decision (figure 4B). (A) Visualization of the influence of input tokens/words on the predicted neural network label compared to the true label for seven examples. Word tokens marked in red are assigned to an emergency and green to a non-emergency, and the greater the opacity of the shading, the stronger the assignment. Our model focuses on words that are related to the symptoms or the mechanism of the event described by the patient and that are most consistent with what would be expected clinically. [CLS] denotes the classification token and [SEP] the separator token in BERT. (B) Shown is the average input feature importance across multiple attribution algorithms for the input to all 49 input neurons of the multilayer perceptron for processing the checkbox questions. The attribution algorithms sometimes disagree in assigning importance scores. However, the most important features correspond to features that would be expected to be clinically relevant to the emergency/non-urgent decision. Discussion A neural network has been developed that can determine the urgency of treatment in an eye ED based on natural language data and a checkbox questionnaire completed by the patient before the human examination. The performance of a combination of both types of data was slightly better than decisions made by the ophthalmologists using either type of data alone. This could assist on-call staff with their decision-making and provide the factors used by the neural network for decision-making, obtained with methods of explainable artificial intelligence. As this is basic research, it is still unclear how the proposed algorithm will be applied as a clinical tool. Although the F1 score demonstrates good performance in determining urgency, there are still false negatives, which can lead to emergency cases being misclassified as non-urgent. Therefore, we suggest that its utilisation be restricted to a supportive tool with human supervision by on-call staff.10 Furthermore, the reasons for the decision obtained using the explainable artificial intelligence methods must be presented alongside the result in order to understand the decision when used in an ED.11 Outside of an ED, it is also conceivable that the tool could be used as a safety net to reduce the number of important cases missed where trained professionals are unavailable.12 Since it can be applied to any natural language, written or spoken, it can be integrated into any digital tool that relies on language. Particularly interesting applications would be outpatient telephone and telemedicine systems,13 smartphone apps14 and online systems, especially those that do not rely on human supervision, such as appointment booking and scheduling systems.15 Input data to the neural network seems to play an important role in decision-making. A person making a decision about the urgency of treatment without a medical examination is provided with many additional inputs compared with our algorithm. These include natural language, the visual appearance of the patient and the macroscopic appearance of the affected eye at presentation, as well as perhaps a questionnaire. In this context, smartphone images have also been described as a potential third multimodal input for diagnosis and triage, with the potential to further improve the performance of our network if added as a third multimodal input.16 Machine learning and deep learning have been successfully used to develop the DOTS triage system, using data from 11 315 patients with good and comparable results to those of our study.6 17 The authors chose a high sensitivity of around 0.95 to avoid missing important cases and accepted a lower specificity for this purpose, which seems ethical. It would be disastrous to miss particularly serious cases requiring urgent treatment, such as perforations or severe chemical burns. In our model, it is certainly possible to opt for a higher sensitivity by varying the threshold for the class probabilities observed from the receiver operating characteristic curve. However, it should be noted that a higher sensitivity would result in more patients needing to be seen immediately. Generalisation and robustness are inherent problems of neural networks. Training data are often limited to specialised problems within a specific setting and are frequently based on retrospective data collected from single institutions.10 If generalisation is poor, the proposed approach may lack scalability to other patient groups.18 Using language itself is associated with reduced generalisation. As this network was trained in German, it is highly probable that it will not be applicable to other languages prior to training with the appropriate data. Furthermore, there are many regional dialects of German, which may influence the decision even within the same language.19 Continuously learning artificial intelligence algorithms would also allow some adaptation to regional language differences, but system performance would need to be continuously monitored.20 In addition to language differences, other factors could influence responses to the open-ended and checkbox questionnaires and thus affect the predictions of the networks, leading to different results, bias and discrimination among patients presenting these factors. These factors include, for example, differences in culture, ethnicity or race, differences in disease incidence and rare cases and gender.21 Furthermore, the proposed system would disadvantage patients who, for medical, intellectual or motor reasons, are unable to complete a written questionnaire or to answer free-response questions. Future studies need to address the lack of generalisability and thus reduce discrimination and bias and improve inclusivity, for example, by training with high-quality and diverse training datasets.22 A neural network is designed to learn an algorithm independently, by providing it with input data and enabling stochastic approximation to reduce errors and continuously improve performance. In this sense, a neural network is a black box due to the lack of transparency in its decision-making process, leading to concerns about trust in the network.11 This is particularly problematic in critical real-world applications where safety and reliability are of concern. For instance, object detectors can be easily deceived by making minor alterations to input images or by generating artificial adversarial examples.11 A self-driving car might fail to detect a stop sign by making minor changes (a few small stickers) to the road sign.23 Understanding the factors that influence the decision in the first place could prevent critical errors.24 Explainable artificial intelligence therefore offers justification, control and the potential for improvement and discovery. It also enables interpretability, explainability, comprehensibility, completeness and causality.23 The calculations of the neural network will still be opaque and untraceable to humans, but the results can be attributed to a component provided to the neural network.25 For example, predictions of non-urgent cases are much less trustworthy than predictions of non-urgent cases with an attribution to the checkbox answers of conjunctivitis, no trauma and a free-form answer of stinging, tearing and a swollen eye. The neural networks were more accurate at identifying emergency cases, while ophthalmologists were better at classifying non-urgent ones. The reasons for these differences are unclear, and we can only speculate about them. The black box nature of the neural network, uneven group sizes and other potential biases are possible reasons.26 Neural networks generally perform better at processing large datasets, identifying subtle patterns and making objective assessments. Clinicians, on the other hand, bring invaluable expertise, judgement and the ability to integrate complex contextual factors. Balancing these two perspectives is essential to maximising the technology’s potential while maintaining the irreplaceable value of human insight in clinical decision-making.27 Our algorithm raises other ethical issues besides those already discussed. We intentionally do not specify an acceptable time for deferring non-urgent cases to a later appointment, as this needs to be discussed in the specific setting in which the algorithm is used, taking into account the available resources.28 A thorough discussion of strict human oversight for control and responsibility for error in this critical setting is needed before the algorithm can be used clinically.29 30 Although the algorithm may support clinical decision-making, ultimate responsibility lies with the human user. In areas with a high volume of decisions, an automation bias can easily occur and be overlooked, whereby humans favour decisions generated by machines and disregard conflicting human decisions or contrary data.31 Promoting risks such as an excessive workload, limitations in time and cognitive resources may be particularly prevalent in an ED, leading to misinterpretation or overlooking information necessary for the clinicians’ decision on urgency.32 Further questions arise regarding the ethics of triage itself, as it is always difficult to withhold care from one patient at the expense of another, but it is also problematic not to treat emergency patients due to lack of resources.33 Other limitations of the study include its retrospective nature and the relatively small number of patients for a neural network approach. This meant that it was not possible to include cases that were completely free of confounders, in which all items of the questionnaire were correctly completed. Higher decision accuracy and better generalisability are expected with more data and larger neural networks, possibly in a multicentre and prospective design to avoid bias and gain more cases. In addition, the questionnaire was not designed for the study or neural network training and was used retrospectively. Better results may be obtained if more specific factors are identified that allow discrimination between emergencies and non-urgent cases. In conclusion, our proposed neural network design could be a valuable addition to initial assessments in ophthalmology EDs. However, future studies should focus on validation in daily clinical practice, improving its accuracy further, integrating it into existing workflows and thoroughly discussing its ethical implications and generalisability.