Journal Article

Large language models for generating medical examinations: systematic review

Nadkarni, Girish, Klang, Eyal, Sorin, Vera, Konen, Eli, Artsi, Yaara, Glicksberg, Benjamin S.

2024

Overview

Background Writing multiple choice questions (MCQs) for the purpose of medical exams is challenging. It requires extensive medical knowledge, time and effort from medical educators. This systematic review focuses on the application of large language models (LLMs) in generating medical MCQs. Methods The authors searched for studies published up to November 2023. Search terms focused on LLMs generated MCQs for medical examinations. Non-English, out of year range and studies not focusing on AI generated multiple-choice questions were excluded. MEDLINE was used as a search database. Risk of bias was evaluated using a tailored QUADAS-2 tool. Results Overall, eight studies published between April 2023 and October 2023 were included. Six studies used Chat-GPT 3.5, while two employed GPT 4. Five studies showed that LLMs can produce competent questions valid for medical exams. Three studies used LLMs to write medical questions but did not evaluate the validity of the questions. One study conducted a comparative analysis of different models. One other study compared LLM-generated questions with those written by humans. All studies presented faulty questions that were deemed inappropriate for medical exams. Some questions required additional modifications in order to qualify. Conclusions LLMs can be used to write MCQs for medical examinations. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed. Until then, LLMs may serve as a supplementary tool for writing medical examinations. 2 studies were at high risk of bias. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.

• Publisher: BioMed Central,BioMed Central Ltd,Springer Nature B.V,BMC
• Subject: Analysis / Artificial intelligence / Authors / Automation / Computational linguistics / Computer Simulation / Curricula / Databases, Factual / Education / Evaluators / Feedback (Response) / Formative evaluation / Generative pre-trained transformer / Humans / Instructional Materials / Internal medicine / Knowledge / Language / Language Processing / Large language models / Medical colleges / Medical Education / Medical examination / Medicine / Meta Analysis / Multiple choice / Multiple choice questions / Multiple choice tests / Multiple-choice examinations / Natural language interfaces / Periodic health examinations / Physical diagnosis / Physical examinations / Standardized tests / Students / Summative Evaluation / Surgery / Systematic review / Technology application / Tests, problems and exercises / Theory of Medicine/Bioethics / Writing