Uncertainty in Standardized Letter of Evaluation Authorship: A Qualitative Study of Emergency Medicine Faculty Authors: Caitlin Schrepel, Eric Shappell, Morgan Sehdev, Nicole M. Dubosh, Benjamin H. Schnapp, Alexis Pelletier‐Bui, Cullen Hegarty, Sharon Bord Published: 2026 ABSTRACT ABSTRACTBackgroundThe standardized letter of evaluation (SLOE) is a cornerstone of emergency medicine (EM) residency selection. While prior research has focused on how SLOEs are interpreted by reviewers, less is known about the considerations and strategies faculty employ when authoring these high‐stakes evaluations.MethodsUsing a constructivist grounded theory approach, we conducted semi‐structured interviews with EM faculty experienced in SLOE authorship between April and October 2025. Interviews were audio‐recorded, transcribed, and iteratively analyzed using constant comparative methods and team‐based reflexive analysis until thematic sufficiency was achieved.ResultsParticipants described SLOE authorship as a process of navigating uncertainty arising from the interaction of structural challenges, cultural pressures, and moral tensions. Structurally, authors described difficulty mapping incomplete or imperfect evaluation data onto standardized SLOE items, often requiring interpretive judgment and use of proxies. Culturally, participants reported pressure to calibrate ratings to perceived national norms to avoid disadvantaging students, contributing to grade inflation and narrative justification strategies. Morally, authors experienced tension between transparency and advocacy, balancing honest assessment with concern for learner success and professional credibility.ConclusionRecognizing uncertainty as a central feature of SLOE authorship underscores the need for improved alignment between clerkship evaluations and SLOE formats, clearer national calibration practices, and faculty development that supports authors in their dual roles as evaluators and mentors. Collectively, these efforts may improve the credibility and consistency of this crucial part of the EM residency application. Introduction The standardized letter of evaluation (SLOE) was designed to streamline the process of evaluating emergency medicine (EM) residency applications by providing a structured assessment of an applicant's clinical performance and perceived readiness for residency [1, 2]. Over the years, the SLOE has been identified as one of the most important tools program directors use when making decisions about interview invites and applicant rankings [3, 4, 5]. Therefore, there has been increasing interest in understanding how faculty reviewers interpret SLOEs and determine competitiveness [6]. Prior studies have highlighted strong consensus among faculty in evaluating SLOE competitiveness, even with the transition to newer competency‐based formats [7, 8, 9]. Additionally, qualitative research has highlighted the complex, multi‐phase process used by reviewers to determine the competitiveness of a SLOE [6]. This work suggests that reviewers prioritize factors such as institutional reputation, ranking distributions, and narrative clarity when assessing SLOE trustworthiness and competitiveness [6]. Although prior work has improved our understanding of how SLOEs are interpreted and used in residency selection, the considerations and strategies authors employ in crafting these evaluations remain underexplored. The limited body of evidence examining SLOE authorship practices reveals disparities in both language and scoring associated with gender, race, and ethnicity [10, 11, 12]. These findings suggest that author‐level decisions may meaningfully shape the evaluation of applicants. Additionally, assessment is increasingly understood as a socially situated, interpretive process, with literature on assessor cognition demonstrating how evaluators interpret imperfect data to make judgments [13, 14, 15]. These perspectives suggest that high‐stakes evaluations such as the SLOE may involve complex judgment processes, yet the nuances involved in SLOE authorship are not yet well characterized. This gap is critical to address, given that the utility of the SLOEs depends not only on reviewers' interpretation but also on the intent, methodology, and transparency of their authorship. Therefore, informed by perspectives on assessor cognition and value‐laden assessment in clinical education [13, 15], our study uses qualitative methods to explore the considerations and strategies of SLOE as they navigate the judgment processes involved in crafting these evaluations. By examining the perspectives and practices of SLOE authors, our research aims to bridge the gap between SLOE design and its practical application, providing insights that can inform future guidelines, training, and tool development for both authors and reviewers. Methods In order to generate a theoretical understanding that captures the intricacies of SLOE authorship, we adopted constructivist grounded theory (CGT) as our methodological approach [16]. We collected data from April to October 2025. Our research team included eight EM faculty members from various institutions across the United States. Team members have held roles in residency, clerkship, fellowship, and institutional leadership. While the team represented diverse institutional contexts, the majority practiced within large academic training programs. Recognizing that our perspectives could influence data collection and analysis, we embedded personal, interpersonal, contextual, and methodological reflexivity throughout the study [17]. For instance, all faculty authors brought experience in writing and interpreting SLOEs and acknowledged that we have our own practices for SLOEs authorship, which could shape how we analyze the data. Similarly, we considered how the context of the institutions where we work may influence our interpretation of the data, given the variability in institutional cultures regarding SLOE authorship. To maintain reflexivity throughout the study, we incorporated meetings for collective reflection to consider how our cognitive and affective responses influenced our analytic decisions. Setting and Participants We recruited EM faculty with a minimum of 1 year of SLOE‐writing experience via the CORD (Council of Residency Directors in Emergency Medicine) listserv. We subsequently expanded our sample through snowball sampling [18] to identify additional participants, thereby ensuring broader representation. We applied purposive sampling [19] to intentionally include faculty with varied levels of SLOE authorship experience, leadership positions, geographic regions, and program lengths, anticipating that these characteristics might influence how they approached SLOE authorship. Data Collection One team member (CS), who has formal training in medical education research, conducted semi‐structured interviews via Zoom (Zoom Video Communications Inc., San Jose, CA). We developed the interview guide (Appendix 1) based on our group's experience authoring SLOEs and insights from our previous work examining SLOE interpretation by reviewers [6]. Our interview guide was also informed by scholarship on assessor cognition and value‐laden assessment [13, 15]. Guided by these perspectives, we sought to understand how SLOE authors made sense of imperfect information and navigated competing expectations when constructing evaluations. Verbal consent to record and analyze de‐identified transcripts was obtained before each interview. Audio recordings were transcribed and de‐identified using Rev. (Rev.com, San Francisco, CA). Using a constant comparative method [20], we refined codes and iteratively updated the interview guide as the study progressed. Data collection continued until thematic sufficiency was achieved [21]. This study was reviewed and deemed exempt by the the University of Washington Human Subjects Division (STUDY00022076). Data Analysis Two investigators (CS, SB) independently coded all transcripts and met after each coding cycle to compare interpretations and resolve discrepancies. To develop the initial codebook, we started by breaking the data into smaller analytic units, which we refined iteratively as additional interviews were analyzed. Axial coding was then used to link codes and categories and build our theoretical framework [20]. Once a preliminary framework was developed, the larger study team reviewed the categorization and interpretations, fostering reflective dialogue and challenging assumptions to enhance the trustworthiness of the data [22]. During later stages of analysis, we drew on scholarship describing uncertainty and assessor cognition in clinical assessment as sensitizing concepts to inform our interpretation of the data [14, 23, 24]. After eight interviews, we determined that our coding structure sufficiently reflected the dataset. We conducted two additional interviews to explore whether any data might challenge our interpretation. As no counterexamples emerged, we concluded that the dataset was sufficient for the study's purpose [21]. Results We interviewed ten EM faculty from institutions representing six of the nine AAMC regions [25] (East North Central, West North Central, South Atlantic, Middle Atlantic, New England, and Pacific), with seven from three‐year EM programs and three from four‐year EM programs. Seven participants currently serve as EM clerkship directors, two hold dual positions as clerkship directors and assistant program directors, and one is a chair emeritus. Experience in authoring SLOEs ranged from 1 year to 20 years, with an average of 6 years of experience. Participants reported encountering multiple challenges and pressures that influenced their approach to authoring SLOEs. These included structural difficulties in mapping evaluation data onto the standardized format, perceived pressure to conform to practices used by peer programs, and internal conflicts between ensuring transparency and maintaining advocacy for students. Ultimately, the work of SLOE authorship emerged as a process of navigating uncertainty created by the structural, cultural, and moral forces (Figure 1). Framework illustrating how SLOE author judgment is influenced by the uncertainty created by the interplay of structural challenges, cultural pressures, and moral tensions. Structural Challenges: Standardizing Incomplete and Imperfect Data Across interviews, participants described having to approximate SLOE ratings when faced with incomplete or imperfect data. In many cases, participants reported that the data available to them from shift evaluations did not align neatly with the SLOE's questions. In these cases, they often relied on proxies, such as overall course grades, faculty impressions, or other clerkship data, to approximate SLOE ratings. For example, if a student was on track to receive an “honors” grade in the EM clerkship, they were frequently considered “fully entrustable” in Part A, even when the specific skills to be evaluated were not directly observed. Several participants also described defaulting to a particular score or institutional norm when data were insufficient, as this participant noted:… I tend to stay towards the straight middle. It's more of a general sense as opposed to each individual criterion, to be honest, because sometimes it gets a little too granular. I feel like I can't answer that to the level of detail that's required. So I try to keep it fair for everyone, and that's how I do it. (05) Programs that aligned their shift evaluations with the SLOE found it easier to complete all SLOE sections. However, regardless of their evaluation structure, most participants found it difficult to determine an applicant's “ability to perform common ED procedures” or “recognize and manage basic emergent situations” in Part A of the SLOE. Many reported limited opportunities to observe students performing procedures and felt students rarely cared for patients ill enough to qualify as basic emergencies, as noted by this participant:[The question on] procedures is difficult, depending on whether the student has performed a lot of procedures. Sometimes I'll only have data on two laceration repairs they've done… and then emergent situations [question] is also a little bit difficult because they're very rarely the ones doing the resuscitation. So I don't know how much they're necessarily managing emergent situations. (06) Participants also emphasized that completing the SLOE required insider knowledge of how individual faculty graded. They described adjusting or filtering scores based on an understanding of who submitted them, acknowledging that “some faculty are just easier graders or harder graders.” (04) Participants highlighted the interpretive work necessary to reconcile imperfect inputs with the expectation of standardization, demonstrating the role of navigating local norms and making subjective judgments in the SLOE authorship process. … I tend to stay towards the straight middle. It's more of a general sense as opposed to each individual criterion, to be honest, because sometimes it gets a little too granular. I feel like I can't answer that to the level of detail that's required. So I try to keep it fair for everyone, and that's how I do it. (05) [The question on] procedures is difficult, depending on whether the student has performed a lot of procedures. Sometimes I'll only have data on two laceration repairs they've done… and then emergent situations [question] is also a little bit difficult because they're very rarely the ones doing the resuscitation. So I don't know how much they're necessarily managing emergent situations. (06) Finally, participants often struggled when asked to rank students in Part C of the SLOE early in the academic year or with limited information outside of their EM clerkship, as explained by this participant:…we are determining our rank list or where we think they're going to fall in our rank list in September when we don't do our rank list until February. So kind of an impossible task…we don't have the full application either, so you're not even dealing with all the information. (03) In navigating these structural challenges, participants emphasized the interpretive work required to author SLOEs and the judgments they made amid imperfect and incomplete information. …we are determining our rank list or where we think they're going to fall in our rank list in September when we don't do our rank list until February. So kind of an impossible task…we don't have the full application either, so you're not even dealing with all the information. (03) Cultural Pressure: Aligning With Other Authors These structural challenges arose within the broader cultural context of SLOE authorship, in which authors understood the SLOE as a comparative tool used by residency programs to evaluate students across institutions. Because of the SLOE's importance, some participants felt pressure to align their approach with that of other SLOE authors. Deviating from perceived common practices—such as routinely using the full distribution of the rating scale for Part A or Part B—was seen as potentially harming their students. Participants described calibrating their ratings not only against internal standards but also against what they believed peer programs were doing. This led to a tendency toward grade inflation, as they feared being the only program that uses the bottom of the scales, as noted by this participant.We struggle with Part B. I know it's supposed to be a bell‐shaped curve, but ours probably doesn't reflect that. I think the first year we did this, it reflected a bell‐shaped curve. Then, when we read SLOEs [from other institutions] throughout the year, and we realized we might've tanked a couple of our students…and so now, I think a lot of our students are going to be around the four. (06) Not all participants succumbed to this perceived pressure. Instead, they emphasized that they continued to use the full scale when it accurately reflected their genuine assessment of a student, even if that meant their distributions looked different from those of their peers. Rather than inflating grades, they used the SLOE narrative or the institutional information in Part E to explain why they scored a student the way they did. While these participants valued “blunt honesty” (07) in their letters, they still felt a need to explain themselves so as not to harm their students, as noted by this participant:I have a clause on the bottom of my SLOE… because I was worried that people would actually say, oh my gosh, they're only a three. So, it says a majority of students will get a three. We follow the bell‐shaped curve. If they get a one or a two or a four, or a five, there will be an explanation as to why. (07) Together, these accounts illustrate how cultural expectations within the national SLOE community shape authors' calibration of their scoring practices, influencing both the distribution of ratings and the narrative strategies they use to justify them. We struggle with Part B. I know it's supposed to be a bell‐shaped curve, but ours probably doesn't reflect that. I think the first year we did this, it reflected a bell‐shaped curve. Then, when we read SLOEs [from other institutions] throughout the year, and we realized we might've tanked a couple of our students…and so now, I think a lot of our students are going to be around the four. (06) I have a clause on the bottom of my SLOE… because I was worried that people would actually say, oh my gosh, they're only a three. So, it says a majority of students will get a three. We follow the bell‐shaped curve. If they get a one or a two or a four, or a five, there will be an explanation as to why. (07) Moral Tension: Balancing Transparency and Advocacy In addition to structural challenges and perceived cultural pressures, participants described the moral tension they felt as they attempted to balance transparency and advocacy. They expressed a sense of responsibility to accurately represent the student on the SLOE while also supporting the student's success in the residency match. Several participants approached this tension by emphasizing their commitment to honesty, framing their candor as essential to maintaining credibility with program directors who rely on the SLOE. These participants reported deliberately including weaknesses and using what they referred to as blunt language, as noted by this participant.And I'm pretty blunt. If [the student] emailed at seven o'clock, calling out for a seven o'clock shift, then it'll say exactly that sentence [in the SLOE]. And that's sometimes perceived as harsh, but I would hate for someone to think that this student was average. When, to me, administrative hygiene, growth mindset, all of these things are far more important than medical knowledge. (07) Ultimately, they felt strongly that this candid language was necessary to protect their own reputation as a SLOE author and to maintain the validity of the SLOE.The last thing I want to do is obstruct a student from getting where they need to go, but I have to balance my reputation… if I'm recommending people who have serious character flaws or teachable flaws…I won't be able to [continue recommending people] if I don't have the proper credibility that's been built throughout years of doing SLOEs. (05) Others admitted to a more protective stance, softening negative feedback or avoiding low ratings altogether in order to shield students. Participants noted that they were very hesitant to give scores lower than a 3 on Parts A and B or a lower ⅓ ranking in Part C of the SLOE. These elections were seen as communicating a red flag, and many participants stated they would only give these scores if there were a clear indication to do so, as described by this participant:We probably don't really use a bell‐shaped curve. I think we probably start with three down the row. But, we're far more likely to give somebody a four, whereas I think giving somebody a two is really signaling a red flag to other programs. And so I think somebody has to really, really do something to warrant getting a two. (04) This advocacy‐oriented approach was framed as important, given the high stakes of residency selection and the disproportionate consequences of a poor rating. Several participants also described modifying their language in the narrative to strike a delicate balance—hinting at concerns without making them explicit, or pairing critiques with strong positives, as noted by this participant:Typically, if, for example, somebody struggles with standardized testing. Then you might say, yes, I think they will require a little extra attention in terms of standardized testing, but then say that your effort will be rewarded because he's really wonderful and has all these other great facets. It is kind of like the shit sandwich: if you've got to say something bad, say something good afterward. (01) Participants acknowledged that these strategies risked undermining the trustworthiness of the SLOE, yet they continued to weigh the impact of disadvantaging students. Therefore, for many, SLOE authorship became an exercise in moral tension, requiring them to navigate the pull between their identity as evaluators committed to transparency and their role as mentors. And I'm pretty blunt. If [the student] emailed at seven o'clock, calling out for a seven o'clock shift, then it'll say exactly that sentence [in the SLOE]. And that's sometimes perceived as harsh, but I would hate for someone to think that this student was average. When, to me, administrative hygiene, growth mindset, all of these things are far more important than medical knowledge. (07) The last thing I want to do is obstruct a student from getting where they need to go, but I have to balance my reputation… if I'm recommending people who have serious character flaws or teachable flaws…I won't be able to [continue recommending people] if I don't have the proper credibility that's been built throughout years of doing SLOEs. (05) We probably don't really use a bell‐shaped curve. I think we probably start with three down the row. But, we're far more likely to give somebody a four, whereas I think giving somebody a two is really signaling a red flag to other programs. And so I think somebody has to really, really do something to warrant getting a two. (04) Typically, if, for example, somebody struggles with standardized testing. Then you might say, yes, I think they will require a little extra attention in terms of standardized testing, but then say that your effort will be rewarded because he's really wonderful and has all these other great facets. It is kind of like the shit sandwich: if you've got to say something bad, say something good afterward. (01) Discussion While prior work has focused on how SLOEs are interpreted [6, 7, 8, 9], our findings demonstrate that SLOE authorship is itself a complex process shaped by structural challenges, cultural pressures, and moral tensions. By outlining these three thematic domains, we offer a framework for understanding how authors make judgments and navigate the uncertainties inherent to SLOE authorship. A central source of this uncertainty stemmed from the structural challenges participants encountered when attempting to complete the SLOE with incomplete or imperfect data. Consistent with scholarship on assessment in clinical education [13, 14], SLOE authors are not simply transcribing data—they are interpreting, filtering, and calibrating it. The SLOE creates the expectation of standardization; however, in reality, clerkships vary in clinical experiences, procedural exposure, and assessment practices. While participants in programs that explicitly aligned their clerkship shift evaluation forms with the SLOE framework faced fewer challenges with this interpretation, they still engaged in data calibration and filtering based on their knowledge of specific faculty scoring habits. Taken together, these dynamics reveal that uncertainty is embedded in the process of completing this seemingly standardized evaluation. Participants also described adjusting their SLOE ratings and narrative in anticipation of how their institution's letters would be perceived relative to those from peer programs. This reflects a form of cultural pressure rooted in uncertainty—specifically, uncertainty about how one's assessment will be interpreted within a competitive, high‐stakes environment such as the residency match. As highlighted in prior work, grade inflation is common throughout medical school clerkships [26, 27, 28, 29], a trend shaped by multiple factors, including the prominent role that clerkship grades play in residency selection [28]. Our study suggests that SLOE authorship practices are influenced by similar cultural calibration pressures, which may undermine the SLOE's intended role as a standardized, comparative assessment. Variability in how institutions interpret and apply rating anchors can reduce reliability and discriminative power, as seen in other grading contexts such as clerkships [30, 31]. Recent work examining the current version of the SLOE found substantial clustering of scores at the upper end of rating scales, raising concerns about the discriminative ability of some SLOE components. These findings align with our participants' descriptions of hesitancy in using the full rating scale and concerns about disadvantaging students by assigning lower scores [32]. As many medical schools adopt pass/fail grading for core clerkships, the relative importance of the SLOE in residency selection has increased, potentially intensifying pressure on authors to align their ratings with perceived norms across peer programs. Finally, the tension between providing an honest assessment and advocating for students reflects the moral tension inherent in SLOE authorship. Assessment, such as SLOE completion, is not a neutral or purely technical act but a value‐laden practice [15, 24] that may threaten the educational alliance by undermining trust [33]. In their dual positioning as both evaluator and mentor, SLOE authors must weigh their obligation to provide an accurate, transparent evaluation against their desire to support a learner's success in the residency match. Prior research on clinician‐educator professional identity formation describes professional identity as an ongoing process of negotiating multiple roles and values [34, 35]. Similarly, the moral tension experienced by SLOE authors in this study represents a negotiation of professional identities in which uncertainty about assessment interpretation forces faculty to reconcile competing professional values, shaping not only how they evaluate learners but also how they understand their role as mentors. Taken together, our findings position uncertainty as a central organizing feature of SLOE authorship, arising from structural data limitations, cultural calibration pressures, and the moral work of value‐laden assessment, and shaping how clinician‐educators negotiate their professional identities as both evaluators and mentors. Our findings suggest several actionable strategies that EM educators should consider as we strive to ensure the SLOE remains relevant for residency selection committees. First, institutions should consider aligning their clerkship evaluation tools more closely with the SLOE format. Doing so would reduce the structural uncertainty authors face by generating more direct, high‐quality data for completing SLOE items, thereby minimizing the need to approximate ratings. Updating tools such as the NCAT [36] to align with the SLOE may also promote greater national alignment in scoring by providing authors with clearer, more consistent evaluative anchors. CORD may also need to reconsider some specific SLOE items—particularly those in Part A addressing procedural competence and performance in emergent situations. Participants noted that these domains are inconsistently observed across clerkship sites or variably emphasized depending on patient mix and supervision patterns. Calibration workshops at national meetings, with examples of SLOEs from various institutions, along with publication of aggregate national distributions of SLOE ratings, may help provide shared reference points and allow authors to compare their institutional scoring patterns with national norms. This could help mitigate variability between institutions, grade inflation, and uncertainty about cultural expectations. Finally, faculty development initiatives should explicitly address the dual role of SLOE authors as both evaluators and mentors. Supporting authors as they navigate their professional identities amid this moral uncertainty may not eliminate the tension between honesty and advocacy. Still, it could provide educators with clearer ethical scaffolding for making these difficult decisions. Looking ahead, several avenues for further research exist that could deepen our understanding of SLOE authorship practices. For example, it would be helpful to see whether programs that align their clerkship evaluation tools with the SLOE format yield letters that reviewers perceive as more trustworthy and easier to interpret. In addition, future work might examine how individual and program‐level interventions (such as author calibration sessions, institutional guidance on SLOE writing, or structured feedback sessions for faculty) affect factors such as rating reliability or narrative depth. Finally, engaging applicants themselves about their perceptions of the SLOE could help triangulate our findings with applicant experience and inform more learner‐centered SLOE reforms. We acknowledge several limitations to consider when evaluating our results. While our sample included authors from six of nine AAMC regions and both 3‐ and 4‐year EM programs, it is possible that we did not capture outlier perspectives. Additionally, although we employed reflexivity and team debriefing to minimize bias, given that many of the study authors have experience writing SLOEs, we acknowledge the potential for insider‐researcher influence on interpretation. Additionally, given the sensitive nature of SLOE authorship, some participants may have been less forthcoming when reflecting on their authorship practices. As with other studies of evaluative practice, participants' narratives may have been influenced by recall and self‐presentation, particularly when reflecting on past decisions that carried significant professional and personal responsibility. Conclusions Our study highlights how structural, cultural, and moral factors generate uncertainty in SLOE authorship, shaping a critical component of the EM residency application. Authors must make judgments while interpreting imperfect data, anticipating peer norms, and balancing transparency with learner advocacy. Aligning clerkship evaluations, supporting calibration, and providing targeted faculty development may reduce uncertainty and strengthen the SLOE's credibility. Author Contributions C.S.: study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, methods expertise, critical revision of the manuscript for important intellectual content. E.S.: study concept and design, analysis and interpretation of the data, drafting of the manuscript, methods expertise, critical revision of the manuscript for important intellectual content, and statistical expertise. M.S.: study concept and design, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. N.M.D.: study concept and design, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. B.H.S.: study concept and design, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. A.P.B.: study concept and design, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. C.H.: study concept and design, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. S.B.: study concept and design, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. Conflicts of Interest The authors declare no conflicts of interest. Supporting information Data S1: aet270175‐sup‐0001‐Supinfo.docx.