Journal Article

Evaluating the accuracy of ChatGPT-4 in predicting ASA scores: A prospective multicentric study ChatGPT-4 in ASA score prediction

Şahin, Ayça Sultan, Turan, Engin İhsan, Baydemir, Abdurrahman Engin, Özcan, Funda Gümüş

2024

Overview

This study investigates the potential of ChatGPT-4, developed by OpenAI, in enhancing medical decision-making processes, particularly in preoperative assessments using the American Society of Anesthesiologists (ASA) scoring system. The ASA score, a critical tool in evaluating patients' health status and anesthesia risks before surgery, categorizes patients from I to VI based on their overall health and risk factors. Despite its widespread use, determining accurate ASA scores remains a subjective process that may benefit from AI-supported assessments. This research aims to evaluate ChatGPT-4's capability to predict ASA scores accurately compared to expert anesthesiologists' assessments. In this prospective multicentric study, ethical board approval was obtained, and the study was registered with clinicaltrials.gov (NCT06321445). We included 2851 patients from anesthesiology outpatient clinics, spanning neonates to all age groups and genders, with ASA scores between I-IV. Exclusion criteria were set for ASA V and VI scores, emergency operations, and insufficient information for ASA score determination. Data on patients' demographics, health conditions, and ASA scores by anesthesiologists were collected and anonymized. ChatGPT-4 was then tasked with assigning ASA scores based on the standardized patient data. Our results indicate a high level of concordance between ChatGPT-4 predictions and anesthesiologists' evaluations, with Cohen's kappa analysis showing a kappa value of 0.858 (p = 0.000). While the model demonstrated over 90% accuracy in predicting ASA scores I to III, it showed a notable variance in ASA IV scores, suggesting a potential limitation in assessing patients with more complex health conditions. The findings suggest that ChatGPT-4 can significantly contribute to the medical field by supporting anesthesiologists in preoperative assessments. This study not only demonstrates ChatGPT-4's efficacy in medical data analysis and decision-making but also opens new avenues for AI applications in healthcare, particularly in enhancing patient safety and optimizing surgical outcomes. Further research is needed to refine AI models for complex case assessments and integrate them seamlessly into clinical workflows. •Introduced ChatGPT-4's capability in accurately predicting American Society of Anesthesiologists (ASA) scores, marking a significant advancement in preoperative patient assessments.•Conducted a prospective multicentric study involving 2851 patients, ensuring a comprehensive evaluation across diverse patient demographics.•Demonstrated high concordance between ChatGPT-4 predictions and expert anesthesiologists’ evaluations, with a notable kappa value of 0.858.•Identified limitations and areas for improvement in AI’s assessment of complex health conditions, particularly in patients with ASA IV scores.•Highlighted the potential of integrating artificial intelligence in enhancing the objectivity and efficiency of preoperative assessments, contributing to safer anesthesia practices and optimized surgical outcomes.

• Publisher: Elsevier Inc,Elsevier Limited
• Subject: Accuracy / Adolescent / Adult / Aged / Aged, 80 and over / Anesthesia / Anesthesia - methods / Anesthesiologists - statistics & numerical data / Anesthesiology - standards / Artificial intelligence / Blood & organ donations / Chatbots / Child / Child, Preschool / Clinical decision making / Clinical Decision-Making - methods / Demographics / Ethics / Female / Gender / Health Status / Humans / Infant / Infant, Newborn / Laboratories / Male / Medical personnel / Medical research / Middle Aged / Pain Medicine / Patient safety / Patients / Perioperative care / Preoperative Care - methods / Preoperative Care - standards / Preoperative Care - statistics & numerical data / Prospective Studies / Reproducibility of Results / Risk Assessment - methods / Risk Assessment - statistics & numerical data / Risk Factors / Surgery / Surgical outcomes / Systemic diseases / Young Adult