Journal Article

Exploring the potential of ChatGPT as an adjunct for generating diagnosis based on chief complaint and cone beam CT radiologic findings

Gao, Antian, Lin, Zitong, Liu, Wenjing, Hu, Yanni, Hu, Ziyang, Wen, Shanhui, Liu, Shu

2024

Overview

Aim This study aimed to assess the performance of OpenAI’s ChatGPT in generating diagnosis based on chief complaint and cone beam computed tomography (CBCT) radiologic findings. Materials and methods 102 CBCT reports (48 with dental diseases (DD) and 54 with neoplastic/cystic diseases (N/CD)) were collected. ChatGPT was provided with chief complaint and CBCT radiologic findings. Diagnostic outputs from ChatGPT were scored based on five-point Likert scale. For diagnosis accuracy, the scoring was based on the accuracy of chief complaint related diagnosis and chief complaint unrelated diagnoses (1–5 points); for diagnosis completeness, the scoring was based on how many accurate diagnoses included in ChatGPT’s output for one case (1–5 points); for text quality, the scoring was based on how many text errors included in ChatGPT’s output for one case (1–5 points). For 54 N/CD cases, the consistence of the diagnosis generated by ChatGPT with pathological diagnosis was also calculated. The constitution of text errors in ChatGPT’s outputs was evaluated. Results After subjective ratings by expert reviewers on a five-point Likert scale, the final score of diagnosis accuracy, diagnosis completeness and text quality of ChatGPT was 3.7, 4.5 and 4.6 for the 102 cases. For diagnostic accuracy, it performed significantly better on N/CD (3.8/5) compared to DD (3.6/5). For 54 N/CD cases, 21(38.9%) cases have first diagnosis completely consistent with pathological diagnosis. No text errors were observed in 88.7% of all the 390 text items. Conclusion ChatGPT showed potential in generating radiographic diagnosis based on chief complaint and radiologic findings. However, the performance of ChatGPT varied with task complexity, necessitating professional oversight due to a certain error rate.

• Publisher: BioMed Central,BioMed Central Ltd,Springer Nature B.V,BMC
• Subject: Accuracy / Artificial Intelligence / CBCT / Chatbots / Complaints / Completeness / Computed tomography / Cone-Beam Computed Tomography / CT imaging / Dental Disease / Dental disorders / Diagnosis / Diagnosis, Computer-Assisted / Diagnostic systems / Errors / Evaluation / Health Informatics / Humans / Information Systems and Communication Service / Language / Large language model / Management of Computing and Information Systems / Medical diagnosis / Medical tests / Medicine / Medicine & Public Health / Neoplastic/cystic diseases / Patients / Performance assessment / Radiologic finding / Radiologic impression / Task complexity / Technology application