ChatG‐PD? Comparing large language model artificial intelligence and faculty rankings of the competitiveness of standardized letters of evaluation
Benjamin Schnapp, Morgan Sehdev, Caitlin Schrepel, Sharon Bord, Alexis Pelletier‐Bui, Al’ai Alvarez, Nicole M Dubosh, Yoon Soo Park, Eric Shappell · 2024
Read ChatG‐PD? Comparing large language model artificial intelligence and faculty rankings of the competitiveness of standardized letters of evaluation by Benjamin Schnapp, Morgan Sehdev, Caitlin Schrepel, Sharon Bord, Alexis Pelletier‐Bui, Al’ai Alvarez, Nicole M Dubosh, Yoon Soo Park, Eric Shappell (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-11628426.txt.
Description
Benjamin Schnapp and co-authors test in 2024 whether an AI chatbot can rank residency application letters. Faculty agree well on which standardized evaluation letters signal strong emergency medicine applicants, but reading them eats hours. The team drafts fifty mock letters and has a data-focused language model rank them seven times.
The model's first consensus ranking matches faculty rankings almost exactly. When the model picks its own criteria and re-ranks, agreement drops somewhat. The authors say such models could grade letters with minimal faculty effort, though their rankings lean on the guidance given. The study weighs automation against judgment in a high-stakes selection step.
Catalog record
| CREATOR | Benjamin Schnapp, Morgan Sehdev, Caitlin Schrepel, Sharon Bord, Alexis Pelletier‐Bui, Al’ai Alvarez, Nicole M Dubosh, Yoon Soo Park, Eric Shappell |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |