Journal Article

A systematic review of the impact of artificial intelligence on educational outcomes in health professions education

Hani, Hind, Feigerlova, Eva, Hothersall-Davies, Ellie

2025

Overview

Background Artificial intelligence (AI) has a variety of potential applications in health professions education and assessment; however, measurable educational impacts of AI-based educational strategies on learning outcomes have not been systematically evaluated. Methods A systematic literature search was conducted using electronic databases (CINAHL Plus, EMBASE, Proquest, Pubmed, Cochrane Library, and Web of Science) to identify studies published until October 1st 2024, analyzing the impact of AI-based tools/interventions in health profession assessment and/or training on educational outcomes. The present analysis follows the PRISMA 2020 statement for systematic reviews and the structured approach to reporting in health care education for evidence synthesis. Results The final analysis included twelve studies. All were single centers with sample sizes ranging from 4 to 180 participants. Three studies were randomized controlled trials, and seven had a quasi-experimental design. Two studies were observational. The studies had a heterogenous design. Confounding variables were not controlled. None of the studies provided learning objectives or descriptions of the competencies to be achieved. Three studies applied learning theories in the development of AI-powered educational strategies. One study reported the analysis of the authenticity of the learning environment. No study provided information on the impact of feedback activities on learning outcomes. All studies corresponded to Kirkpatrick’s second level evaluating technical skills or quantifiable knowledge. No study evaluated more complex tasks, such as the behavior of learners in the workplace. There was insufficient information on training datasets and copyright issues. Conclusions The results of the analysis show that the current evidence regarding measurable educational outcomes of AI-powered interventions in health professions education is poor. Further studies with a rigorous methodological approach are needed. The present work also highlights that there is no straightforward guide for evaluating the quality of research in AI-based education and suggests a series of criteria that should be considered. Trial registration Methods and inclusion criteria were defined in advance, specified in a protocol and registered in the OSF registries ( https://osf.io/v5cgp/ ). Clinical Trial number: not applicable.

• Publisher: BioMed Central,BioMed Central Ltd,Springer Nature B.V,BMC
• Subject: Accuracy / AI-based training and assessment / Algorithms / Allied Health Occupations Education / Artificial Intelligence / Authors / Chatbots / Communication / Communication Skills / Competence / Computer Simulation / Database Management Systems / Deep learning / Education / Educational aspects / Educational Assessment / Educational Change / Educational evaluation / Educational Methods / Educational Needs / Educational Objectives / Educational research / Educational Strategies / Educational Technology / Educational outcomes / Emergency medical care / Feedback / Feedback (Response) / Generative artificial intelligence / Health Occupations / Health Occupations - education / Health professions / Human performance / Humans / Influence of Technology / Information Seeking / Instructional Effectiveness / Intelligent Tutoring Systems / Language / Language Processing / Large language models / Learning / Learning Analytics / Learning theories and principles / Machine learning / Medical Education / Medical Evaluation / Medical personnel / Medical records / Medical students / Natural Language Processing / Neural networks / Outcomes of Education / Outcomes of Treatment / Participant Characteristics / Participant Satisfaction / Performance evaluation / Personalized learning / Professions / Technology application / Theory of Medicine/Bioethics