Performance of large language models on the Turkish Pharmacy Specialty Examination: a comparative analysis of accuracy, confidence, and readability
Ugur Balaban, Bahar Erbay, Mustafa Aktan, Kutay Demirkan · 2026
Read Performance of large language models on the Turkish Pharmacy Specialty Examination: a comparative analysis of accuracy, confidence, and readability by Ugur Balaban, Bahar Erbay, Mustafa Aktan, Kutay Demirkan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-13482615.txt.
Description
Balaban and colleagues sit three chatbots for Turkey's Pharmacy Specialty Examination in 2026, grading not just their answers but their confidence and the readability of their explanations. The test used 84 publicly available multiple-choice questions drawn from exam years 2017 through 2025, each asked in a fresh session with the same prompt. The three models, ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash, all attempted the full set.
The highest score went to Gemini 2.5 Flash at 92.9 percent correct, trailed by ChatGPT-5.1, which reached 90.5 percent, and DeepSeek-R1 at 89.3 percent, differences the statistics call insignificant overall. Confidence was the worrying part, since the models stamped maximum confidence on most of their wrong answers, ChatGPT-5.1, which hit 87.5 percent of its errors in this test. For readability, the plainest explanations came from ChatGPT-5.1, which scored lowest on two complexity indices, and the authors allow classroom use only under expert supervision.
Catalog record
| CREATOR | Ugur Balaban, Bahar Erbay, Mustafa Aktan, Kutay Demirkan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |