Journal Article

Enhancing responses from large language models with role-playing prompts: a comparative study on answering frequently asked questions about total knee arthroplasty

Chen, Yi-Chen, Hu, Chih-Chien, Sheu, Huan, Lin, Sheng-Hsuan, Yang, Cheng-Pang, Lee, Sheng-Hsun, Lin, Yu-Chih, Fu, Shih-Chen

2025

Overview

Background The application of artificial intelligence (AI) in medical education and patient interaction is rapidly growing. Large language models (LLMs) such as GPT-3.5, GPT-4, Google Gemini, and Claude 3 Opus have shown potential in providing relevant medical information. This study aims to evaluate and compare the performance of these LLMs in answering frequently asked questions (FAQs) about Total Knee Arthroplasty (TKA), with a specific focus on the impact of role-playing prompts. Methods Four leading LLMs—GPT-3.5, GPT-4, Google Gemini, and Claude 3 Opus—were evaluated using ten standardized patient inquiries related to TKA. Each model produced two distinct responses per question: one generated under zero-shot prompting (question-only), and one under role-playing prompting (instructed to simulate an experienced orthopaedic surgeon). Four orthopaedic surgeons evaluated responses for accuracy and comprehensiveness on a 5-point Likert scale, along with a binary measure for acceptability. Statistical analyses (Wilcoxon rank sum and Chi-squared tests; P  < 0.05) were conducted to compare model performance. Results ChatGPT-4 with role-playing prompts achieved the highest scores for accuracy (3.73), comprehensiveness (4.05), and acceptability (77.5%), followed closely by ChatGPT-3.5 with role-playing prompts (3.70, 3.85, 72.5%, respectively). Google Gemini and Claude 3 Opus demonstrated lower performance across all metrics. In between-model comparisons based on zero-shot prompting, ChatGPT-4 achieved significantly higher scores of both accuracy and comprehensiveness relative to Google Gemini ( P  = 0.031 and P  = 0.009, respectively) and Claude 3 Opus ( P  = 0.019 and P  = 0.002), and demonstrated higher acceptability than Claude 3 Opus ( P  = 0.006). Within-model comparisons showed role-playing significantly improved all metrics for ChatGPT-3.5 ( P  < 0.05) and acceptability for ChatGPT-4 ( P  = 0.033). No significant prompting effects were observed for Gemini or Claude. Conclusions This study demonstrates that role-playing prompts significantly enhance the performance of LLMs, particularly for ChatGPT-3.5 and ChatGPT-4, in answering FAQs related to TKA. ChatGPT-4, with role-playing prompts, showed superior performance in terms of accuracy, comprehensiveness, and acceptability. Despite occasional inaccuracies, LLMs hold promise for improving patient education and clinical decision-making in orthopaedic practice. Clinical trial number Not applicable.

• Publisher: BioMed Central,BioMed Central Ltd,Springer Nature B.V,BMC
• Subject: Acceptability / Accuracy / Arthroplasty (knee) / Arthroplasty, Replacement, Knee / Artificial Intelligence / Chatbots / Clinical decision making / Clinical medicine / Comparative studies / Decision making / Decision support systems / Education / Ethical Issues on the Application of Artificial Intelligence to Health and Medicine / Health Informatics / Humans / Information Systems and Communication Service / Joint replacement surgery / Knee / Language / Large Language Models / Management of Computing and Information Systems / Medical education / Medicine / Medicine & Public Health / Methods / Orthopedics / Patient education / Patients / Performance enhancement / Prompt engineering / Prompting / Questions / Role playing / Statistical analysis / Surgeons / Technology application / Total knee arthroplasty