Journal Article

Comparative evaluation of responses from DeepSeek-R1, ChatGPT-o1, ChatGPT-4, and dental GPT chatbots to patient inquiries about dental and maxillofacial prostheses

Özcan, Berna, Özcivelek, Tuğgen

2025

Overview

Background Artificial intelligence chatbots have the potential to inform and guide patients by providing human-like responses to questions about dental and maxillofacial prostheses. Information regarding the accuracy and qualifications of these responses is limited. This in-silico study aimed to evaluate the accuracy, quality, readability, understandability, and actionability of the responses from DeepSeek-R1, ChatGPT-o1, ChatGPT-4, and Dental GPT chatbots. Methods Four chatbots were queried about 35 of the most frequently asked patient questions about their prostheses. The accuracy, quality, understandability, and actionability of the responses were assessed by two prosthodontists using five-point Likert scale, Global Quality Score, and Patient Education Materials Assessment Tool for Printed Materials scales, respectively. Readability was scored using the Flesch-Kincaid Grade Level and Flesch Reading Ease. The agreement was assessed using the Cohen Kappa test. Differences between chatbots were analyzed using the Kruskal-Wallis test, one-way ANOVA, and post-hoc tests. Results Chatbots showed a significant difference in accuracy and readability ( p  <.05). Dental GPT recorded the highest accuracy score, whereas ChatGPT-4 had the lowest. DeepSeek-R1 performed best, while Dental GPT had the lowest performance in readability. Quality, understandability, actionability, and reader education scores did not show significant differences. Conclusions While accuracy may vary among chatbots, the domain-specific trained AI tool and ChatGPT-o1 demonstrated superior accuracy. Even if accuracy is high, misinformation in health care can have significant consequences. Enhancing the readability of the responses is essential, and chatbots should be chosen accordingly. The accuracy and readability of information from chatbots should be monitored for public health.

• Publisher: BioMed Central,BioMed Central Ltd,Springer Nature B.V,BMC
• Subject: Access to information / Accuracy / Artificial Intelligence / Caregivers / Chatbots / Comparative analysis / Comprehension / Computational linguistics / Datasets / Dental Prosthesis / Dental research / Dentistry / Dentists / Dentures / Digital health / Evaluation / Generative Artificial Intelligence / Geriatrics / Health literacy / Humans / Implants, Artificial / Language processing / Large language models / Maxillofacial / Maxillofacial Prosthesis / Medicine / Natural language interfaces / Older people / Oral and Maxillofacial Surgery / Oral hygiene / Patient education / Patient Education as Topic - methods / Patients / Prostheses / Prosthesis / Prosthetics / Public health / Queries / Readability / Social aspects / Surgery / Teeth / Trauma