Automated Generation and Human Evaluation of Neurosurgical Board Examination Self-Assessment Questions Authors: Anton Alyakin, Jaden Stryker, Daniel Alexander Alber, Jin Vivian Lee, Shrutika Singh, Akshay Save, David Kurland, Cordelia Orillac, Aly A. Valliani, Sean Neifert, Darryl Lau, Ilya Laufer, Peter A. Rozman, Eveline Teresa Hidalgo, Howard Riina, Eric C. Leuthardt, Douglas Kondziolka, Laura Snyder, Eric Karl Oermann Published: 2026 ABSTRACT BACKGROUND AND OBJECTIVES:Multiple-choice questions are the primary assessment format for neurosurgical board certification. Creating high-quality examination questions requires significant expert time and resources. The goal of this study was to develop an automated system to generate board-style neurosurgical multiple-choice questions using state-of-the-art vision-language models and compare their quality with authentic self-assessment questions.METHODS:We developed an automated pipeline using OpenAI generative pre-trained transformer (GPT)-4o and Anthropic Claude Sonnet-3.5 to generate neurosurgical board-style questions from Neurosurgery Publications articles. We generated 89 587 synthetic questions: 45 689 with GPT-4o and 43 898 with Claude. Each question was associated with a single image extracted from the articles' figures. We evaluated the quality of synthetic questions through 5 surveys comparing 20 synthetic questions (10 from each model) with 10 authentic questions from the Self-Assessment for Neurological Surgeons (SANS) question bank. Each survey was completed by a neurosurgery resident and an attending who guessed the source [human vs artificial intelligence (AI)-generated] and rated suitability for board examination use. We also evaluated the question-answering performance of the generalist GPT-4o and the specialized CNS-Obsidian.RESULTS:SANS questions were more often perceived as human-made than GPT-generated (residents, P = .0002; attendings, P = .1091) and Claude-generated (residents, P = .0002; attendings, P = .0272) questions. Notably, 54% of AI-generated questions misled at least one evaluator, and 23% misled both. In quality assessments, SANS questions outperformed GPT-generated (residents, P < 10−5; attendings, P = .0001) and Claude-generated (residents and attendings, P < 10−5) questions. Particularly, 25% of AI-generated questions were rated as suitable for board examinations vs 72% of human-generated questions when measured by evaluator consensus (P < 10−7).CONCLUSION:Although quality gaps exist between AI-generated and human-created neurosurgical board examination questions, our approach demonstrates the potential of vision-language models to augment assessment development in specialized medical fields, reducing the burden on examination boards and credentialing organizations. ABBREVIATIONS: artificial intelligence application programming interface generative pre-trained transformer large language model multiple-choice questions postgraduate year Self-Assessment in Neurological Surgery. Multiple-choice questions constitute the primary assessment method for evaluating medical knowledge and clinical reasoning across specialties, forming the foundation of standardized examinations from medical school through board certification.123 In neurosurgery, the American Board of Neurological Surgery requires residents to pass a comprehensive written primary examination before their chief year.4 To prepare for these examinations, residents use practice resources, with the Congress of Neurological Surgeons' Self-Assessment in Neurological Surgery question bank serving as the predominant study tool, containing thousands of board-style questions across all neurosurgical subspecialties.5,6 The traditional development of high-quality medical multiple-choice questions (MCQs) requires significant investment across multiple dimensions. Time requirements are substantial, with healthcare professionals or medical educators typically requiring up to 1 hour to develop a single high-quality MCQ.7 This translates to considerable labor costs, particularly given the expertise required—board-certified physicians with both clinical knowledge and pedagogical understanding must craft questions that accurately assess competency.8 The financial costs are substantial: individual test items cost between $1500 and $2500 to develop, with comprehensive item banks for computer-adaptive testing potentially requiring $3 to $5 million in investment.9 For robust computer-based testing, Breithaupt et al10 calculated that a 40-item computerized examination administered twice yearly requires at least 2000 items to maintain test security and validity. These resource-intensive requirements have prompted medical educators to explore automated alternatives, particularly large language models that could potentially generate educational content at scale while reducing both time and financial burdens.1 Recent studies have explored using large language models to generate medical education content, with varying success across specialties. A comprehensive review of 23 studies found that generative pre-trained transformer (GPT)-generated MCQs showed mixed validity evidence, with 16% to 43% requiring revision and discrimination levels ranging from poor to acceptable.2 Specific implementations have included generating 50 MCQs for internal medicine and surgery examinations, where artificial intelligence (AI)-generated questions scored 7.56 vs 7.88 for human-written questions in relevance,11 creating dermatology board-style questions with only 40% deemed suitable for examination preparation,12 and producing 104 radiology MCQs that matched human-written quality.1 However, these studies focused exclusively on text-based question generation without incorporating visual content, generated relatively small question sets (typically <200), and did not leverage domain-specific medical literature as source material. This study investigates whether current vision-language models can generate multiple-choice questions from neurosurgical literature by leveraging a previously developed pipeline for extracting figure-caption pairs from neurosurgical journals.13,14 We generated 89 587 AI-generated questions from figures in Neurosurgery, Operative Neurosurgery, and Neurosurgery Practice journals using both GPT-4o (OpenAI, Inc.) and Claude Sonnet-3.5 (Anthropic PBC). By comparing a subset of these synthetic questions against authentic Congress of Neurological Surgeons' Self-Assessment in Neurological Surgery questions through blinded expert evaluation, we aimed to determine whether AI-generated visual questions could serve as supplements to existing board preparation materials. Understanding the capabilities and limitations of automated question generation from domain-specific visual content could expand educational resources in neurosurgical training. METHODS Pre-approvals This study was approved by Neurosurgery Publications. No Institutional Review Board or ethics committee approval was required because we used Neurosurgery Publications manuscripts. This study did not involve patients. Data Source and Extraction We extracted data from 3 neurosurgical journals: Neurosurgery (n = 21 219 articles), Operative Neurosurgery (n = 2618 articles), and Neurosurgery Practice (n = 147 articles). Amazon Textract (Amazon Web Services, Inc.) detected figure locations and generated bounding boxes for cropping. A regex-based caption matcher paired images with their corresponding captions using geometric distance calculations. Text extraction used Amazon Textract and Nougat (Meta Platforms, Inc.) to identify in-text figure mentions, linking them to corresponding figure captions. This pipeline has been validated in prior work developing domain-specific Data sets and models for neurosurgical applications.13-15 Image Classification We built an image content classification system using a ResNet-50-based classifier to categorize 500 manually annotated figures. For MCQ generation, we used 2 classes: Class A (medical imaging, such as x-ray, computed tomography, and MRI, and clinical visuals, such as surgical fields, microscopy, anatomic drawings) and Class B (technical content, such as flowcharts and tables). The classifier was trained on 400 images and validated on 100 held-out images. We applied this classifier to the full Data set and only used figures classified as Class A for question generation. MCQ Generation Pipeline We developed an automated pipeline using GPT-4o and Claude Sonnet-3.5 to convert specialized domain knowledge into visual multiple-choice question data sets (Figure 1). Models were prompted to create a multiple-choice question with a discussion based on the image, caption, and in-text mention (Supplemental Digital Content 1, Figure S1, http://links.lww.com/NS9/A122). Models were instructed to avoid explicit references to the existence of the caption and the in-text mention. The application programming interface (API) call also included 4 randomly sampled few-shot examples from a pool of 10 manually created conversion examples. The pipeline included the image of the target figure as a byte string in the API call, but excluded example figures. We used structured JavaScript Object Notation outputs and performed manual post hoc cleaning. Knowledge translation pipeline demonstrated through conversion of neurosurgical data into training Data sets. The example illustrates the automated conversion of specialized knowledge (figure, caption, in-text mention) into the clinical vignette with multiple-choice options and detailed explanation data. The pipeline uses large language models (GPT-4o here) with few-shot in-context learning (4 examples for each task randomly selected from pools of 10) to generate consistently formatted outputs while preserving diagnostic accuracy and educational value. Caption and in-text mention written by the authors. Figure illustration: Created in BioRender. Alyakin, A. (2026) https://BioRender.com/m88wxmt. Figure: Created in BioRender. Alyakin, A. (2026) https://BioRender.com/lvfkxs1. MCQ Quality Evaluation Five surveys were created with 30 questions each: 10 authentic Self-Assessment in Neurological Surgery (SANS) questions, 10 GPT-generated questions, and 10 Claude-generated questions, presented in blinded, randomized order. Each survey was independently evaluated by one neurosurgery resident (PGY-2, PGY-4, PGY-5, PGY-6, and PGY-7) and one attending neurosurgeon. Only image-associated questions from the SANS question bank were included to maintain consistency with our image-based synthetic Data sets. No further filtering was applied to synthetic questions selected. For each survey question, evaluators answered 2 follow-up questions: (1) “Was the question human- or AI-generated?”, and (2) “Was it suitable for neurosurgery board preparation?”. We also evaluated the question-answering performance of the generalist GPT-4o and the specialized CNS-Obsidian15 AI models on these samples. Statistical Analysis Two-sided Fisher's exact tests were performed to compare differences in proportions between models for perceived authenticity and question quality. Consensus quality was defined as both evaluators rating a question as suitable. Paired comparisons between resident and attending performance were assessed using McNemar's test. Outcomes tested included (1) accuracy in answering questions, (2) ability to correctly identify question source (AI vs human), and (3) rating a question as suitable for board examination use. In addition, we performed linear regression analyses to test whether evaluator experience level [resident postgraduate year (PGY) or attending years in practice] predicted performance in 3 aforementioned domains. RESULTS Question Generation Output A total of 89 587 synthetic MCQ questions were generated from the Data set. Human Perception of Question Origin SANS questions were more often perceived as human-made than GPT-generated questions (residents P = .0002; attendings P = .1091) and Claude-generated questions (residents P = .0002; attendings P = .0272). Overall, 54% of AI-generated questions misled at least one evaluator into thinking they were human-made, and 23% misled both evaluators (Figure 2). Multiple choice human evaluation. Thirty-question forms (10 GPT-generated, 10 Claude-generated, 10 board examination question bank—SANS) were distributed to residents and attendings to evaluate accuracy and compare human performance with vision-language models (GPT-4o and CNS-Obsidian). A-C, Accuracy: vision-language model and human performance on GPT-generated, Claude-generated MCQs, and self-assessment question samples. The GPT-generated MCQs and Claude-generated MCQs consist of questions excluded from the CNS-Obsidian training data. D, Human identification of question origin: participants guessed whether questions were from the self-assessment bank or AI-generated. SANS questions were more often perceived as human-made than GPT-generated (residents, P = .0002; attendings, P = .1091) and Claude-generated (residents, P = .0002; attendings, P = .0272) questions for both groups. Notably, 54% of AI-generated questions misled at least 1 evaluator, and 23% misled both. E, Question quality ratings: evaluators rated questions as suitable for neurosurgery board exams. SANS questions outperformed GPT-generated (residents, P < 10−5; attendings, P = .0001) and Claude-generated (residents and attendings, P < 10−5) questions. Pooled together, SANS outperformed AI-generated questions if measured by consensus of the reviewers (requiring both evaluators to mark the question as good, P < 10−7). AI, artificial intelligence; CNS, Congress of Neurological Surgeons; GPT, generative pre-trained transformer; MCQs, multiple-choice questions; SANS, Self-Assessment of Neurological Surgery. Question Quality Assessment When evaluators rated questions as suitable for board examinations, SANS questions significantly outperformed GPT-generated (residents P < 10−5; attendings P = .0001) and Claude-generated questions (residents and attendings, P < 10−5). Overall, 25% of AI-generated questions were rated as high-quality board review questions vs 72% of human-generated questions (P < 10−7). Effect of Evaluator Experience Paired comparisons of resident vs attending responses using McNemar's test revealed no significant differences in accuracy (P = .3916) or source attribution (P = .2327). A significant difference was observed for quality ratings, with residents more likely than attendings to rate questions as suitable (P = .0019). This effect disappeared for SANS and Claude- and nearly disappeared (P = .049) for GPT-generated questions when stratified by the question source (Supplemental Digital Content 1, Table S1, http://links.lww.com/NS9/A122). Linear regression analyses of evaluator experience demonstrated no statistically significant associations with performance across the 3 measured domains (Figure 3). Regression analyses of evaluator experience. Linear regression models tested whether resident PGY or attending years in practice (including residency) predicted A, accuracy in answering questions, B, ability to identify question origin, and C, likelihood of rating a question as suitable. No statistically significant associations were observed in either group. AI, artificial intelligence; GPT, generative pre-trained transformer; PGY, postgraduate year; SANS, Self-Assessment in Neurological Surgery. Impact of Image Quality Subjective reviewer comments noted that images used in AI-generated questions were often less suited for board-style testing. Review of specific examples showed that questions using appropriate clinical images could achieve high-quality ratings and deceive evaluators. By contrast, those with poorly matched images were easily identified as AI-generated and rated as unsuitable (Figure 4). We performed further analysis trying to identify the root of suboptimal questions as older studies (Supplemental Digital Content 1, Figure S2, http://links.lww.com/NS9/A122) or specific topics (Supplemental Digital Content 1, Table S2, http://links.lww.com/NS9/A122). Although the quality had consistent upward trends over time, the regression was not statistically significant. The effect of figure quality on the reviewer perception of AI-generated neurosurgical board examination-like questions. A, AI-generated question based on a figure from “Certain Autobiographical Impressions in the Life of a Neurosurgeon: Chapter 3” (1972, Neurosurgery).16 B, An AI-generated question based on a figure from “Theoretical Aspects of Radiofrequency Lesions in the Dorsal Root Entry Zone” (1984, Neurosurgery).17 One of the reviewers of this question wrote “cat brain!?” across the evaluation form. C, An example of an excellent AI-generated board-like question ranked highly and misleading both reviewers to think it is genuine self-assessment question. “Posterior Fossa Aneurysms: Chapter XV” (1965, Neurosurgery).18 We conjecture that the quality of questions, as ranked by reviewers, was often more dependent on the image not being suitable for the boards rather than the performance of the AI model. AI, artificial intelligence; DREZ, dorsal root entry zone. Figure panels used with permission as follows: Figure panel A16 Gurdjian et al, © Wolters Kluwer; Figure panel B17 Cosman et al, © Wolters Kluwer; Figure panel C18 Logue et al, Wolters Kluwer. AI Model Performance On synthetic questions, GPT-4o achieved 84% accuracy on GPT-generated and 68% on Claude-generated questions, whereas CNS-Obsidian scored 84% and 70%, respectively. Human performance was lower, with residents scoring 58% on GPT-generated and 54% on Claude-generated questions. On authentic SANS questions, GPT-4o achieved 74%, CNS-Obsidian scored 54%, whereas human residents and attendings performed at 70% and 72%, respectively (Figure 2). DISCUSSION We present a method to automate the generation of neurosurgical board examination questions using AI and assess the pearls and pitfalls of our approach to guide future development of synthetic educational materials. Furthermore, compared with attempts in dermatology,12 radiology,1 and general medicine11 where large language models (LLMs) were manually prompted for each question, we created a fully automated pipeline for generating multimedia questions conditioned on figures and captions sourced from scientific publications. Conditioning LLMs on specific excerpts from scientific literature solves 2 critical challenges simultaneously. First, it prevents the model from defaulting to generic, predictable questions that cluster around common neurosurgical topics—a typical outcome when prompted without specific source material or request. Second, while asking an LLM to generate questions about specialized neurosurgical knowledge often leads to fabricated or inaccurate content,19 anchoring the model to peer-reviewed publications dramatically reduces such hallucinations. By grounding question generation in verified scientific text and figures, the model must work within the constraints of accurate information rather than attempting to reconstruct potentially obscure medical facts from its training data. This approach ensures both diversity in question topics and factual accuracy, addressing the dual concerns of educational variety and medical precision. Comparison With Previous Studies Our 25% suitability rate aligns with emerging literature on AI-generated medical MCQs. A systematic review of 23 studies found 16% to 43% of GPT-generated MCQs required revision.2 Other studies reported similarly modest success rates, with only 20% and 40% of AI-generated questions meeting accuracy and appropriateness standards in general surgery and dermatology.12,20 Our findings are consistent with the literature but uniquely demonstrate additional challenges when incorporating visual content—reviewer comments frequently identified image-question misalignment as the primary quality determinant. Economic Implications and Efficiency Human MCQ development can cost up to $1500 to $2500 per item, with comprehensive test banks requiring $3 to $5 million investment.9,21 Our entire pipeline generated 89 587 questions for approximately $1600 in computational costs, less than the traditional cost of developing a single question. Even accounting for our 25% quality rate, this yields approximately 22 400 potentially usable questions at $0.07 per question, representing a 20 000-fold cost reduction. Creating a single high-quality MCQ requires up to 1 hour of a healthcare professional's time,7 while using the online ChatGPT (OpenAI, Inc.) interface, one can generate 50 questions in 20 minutes11 Our automated pipeline, operating at approximately 4 seconds per API call, could generate all 89 587 questions in roughly 100 hours if run sequentially. Through parallel processing, the entire generation was completed in <2 hours—a time frame that would require approximately 6000 to 90 000 hours for human experts.7,11 This 3000 to 45 000-fold time reduction demonstrates how AI can transform resource-intensive educational content creation into a scalable process, potentially enabling rapid updates to question banks as medical knowledge evolves. Proposed Future Framework Importantly, our cost analysis does not account for the human vetting required to identify the 25% of suitable questions from the candidate pool. Although computational generation is inexpensive, expert evaluation remains a bottleneck. Inspired by recent advances in fully automated AI research systems such as The AI Scientist22 (Sakana AI Co., Ltd.) and Otto-SR,23 which demonstrate that AI can effectively review science and improve AI-generated content, we propose a future multiagent AI framework to address the quality gap before human evaluation:Preprocessing agent: Screens source literature and figures to exclude outdated studies, poorly illustrative images, or content unlikely to generate high-quality questions. Ensures that the input pool is relevant, pedagogically appropriate, and aligned with current neurosurgical practice.Question generation agent (this work): Converts preprocessed figures, captions, and in-text mentions into multiple-choice candidate questions using vision-language models.Quality assessment agent: Automatically evaluates each question for factual accuracy, clarity, alignment with source material, and cognitive difficulty.Peer-review agent: Compares candidate questions for redundancy, coverage across topics, and consistency with board-style expectations.Ranking and filtering agent: Ranks questions by predicted suitability and flags top candidates for optional human review. Preprocessing agent: Screens source literature and figures to exclude outdated studies, poorly illustrative images, or content unlikely to generate high-quality questions. Ensures that the input pool is relevant, pedagogically appropriate, and aligned with current neurosurgical practice. Preprocessing agent: Screens source literature and figures to exclude outdated studies, poorly illustrative images, or content unlikely to generate high-quality questions. Ensures that the input pool is relevant, pedagogically appropriate, and aligned with current neurosurgical practice. Question generation agent (this work): Converts preprocessed figures, captions, and in-text mentions into multiple-choice candidate questions using vision-language models. Question generation agent (this work): Converts preprocessed figures, captions, and in-text mentions into multiple-choice candidate questions using vision-language models. Quality assessment agent: Automatically evaluates each question for factual accuracy, clarity, alignment with source material, and cognitive difficulty. Quality assessment agent: Automatically evaluates each question for factual accuracy, clarity, alignment with source material, and cognitive difficulty. Peer-review agent: Compares candidate questions for redundancy, coverage across topics, and consistency with board-style expectations. Peer-review agent: Compares candidate questions for redundancy, coverage across topics, and consistency with board-style expectations. Ranking and filtering agent: Ranks questions by predicted suitability and flags top candidates for optional human review. Ranking and filtering agent: Ranks questions by predicted suitability and flags top candidates for optional human review. We envision the future final product—“AI Neurosurgery Tutor”—as a flexible system that combines some or all of these agents. Importantly, incorporating a Preprocessing Agent would reduce the generation of clearly unsuitable items—the major pitfall of the current include-all system (Figure 4). Limitations Most importantly, despite generating 89 587 questions, we were able to conduct manual evaluation of only 100 synthetic questions, 50 per model. Although our evaluation sample size aligns with previous studies1,11 and implements rigorous blinded methodology, the discrepancy between generation scale and evaluation scale highlights a fundamental challenge: computational generation is easily scalable, but human expert validation remains a critical bottleneck. Our evaluator pool was limited to 10 neurosurgical experts—5 residents, 5 attendings—from a single institution. The practical challenges of recruiting neurosurgeons, both residents and attendings, for educational research—given their clinical obligations—constrains the ability to achieve larger-scale validation. The temporal span of our source material (including articles from as early as 1960s) may have introduced outdated clinical scenarios, particularly for evolving treatment paradigms, although this would less affect anatomy or basic imaging interpretation questions. The automated figure extraction pipeline may have captured figures not intended for educational use.13,14 Despite applying ResNet-50 classification to filter images, some figures found in scientific journals are inherently unsuitable for board-style questions (eg, historical photographs, technical diagrams, basic graphs) survived the filtering, which may have contributed to the image-question misalignment issues identified by reviewers. Importantly, all of the limitations above present opportunities for introducing further filtering, tiering, or refinement using agent-based feedback loops where AI reviews the outputs of AI before expert human evaluation. We restricted our scope to single-image questions and did not investigate text-only or multi-panel imaging studies common in board examinations. Finally, our source material was limited to 3 neurosurgery-specific journals, excluding other educational resources such as Radiopaedia24 (Radiopaedia.org Pty Ltd) or CNS-Nexus (Congress of Neurological Surgeons)25 that might provide more diverse or pedagogically oriented visual content. CONCLUSION This study evaluated vision-language models for generating neurosurgical board questions, finding that only 25% of AI-generated questions matched the quality of human-created ones (72% suitability). Despite this quality gap, the approach achieved remarkable efficiency, generating 89 587 candidate questions for less than the cost of one traditional MCQ. Image-question alignment emerged as the critical quality determinant. Although AI cannot yet replace human expertise for high-stakes assessments, these tools show promise for supplementing educational resources and democratizing access to practice materials in specialized medical fields. Supplementary Material