Journal Article

ChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results

Hristidis, Vagelis, Ganta, Sai Rithesh Reddy, Brown, Ellen L, Stewart, Selena, Ruggiano, Nicole

2023

Overview

People living with dementia or other cognitive decline and their caregivers (PLWD) increasingly rely on the web to find information about their condition and available resources and services. The recent advancements in large language models (LLMs), such as ChatGPT, provide a new alternative to the more traditional web search engines, such as Google. This study compared the quality of the results of ChatGPT and Google for a collection of PLWD-related queries. A set of 30 informational and 30 service delivery (transactional) PLWD-related queries were selected and submitted to both Google and ChatGPT. Three domain experts assessed the results for their currency of information, reliability of the source, objectivity, relevance to the query, and similarity of their response. The readability of the results was also analyzed. Interrater reliability coefficients were calculated for all outcomes. Google had superior currency and higher reliability. ChatGPT results were evaluated as more objective. ChatGPT had a significantly higher response relevance, while Google often drew upon sources that were referral services for dementia care or service providers themselves. The readability was low for both platforms, especially for ChatGPT (mean grade level 12.17, SD 1.94) compared to Google (mean grade level 9.86, SD 3.47). The similarity between the content of ChatGPT and Google responses was rated as high for 13 (21.7%) responses, medium for 16 (26.7%) responses, and low for 31 (51.6%) responses. Both Google and ChatGPT have strengths and weaknesses. ChatGPT rarely includes the source of a result. Google more often provides a date for and a known reliable source of the response compared to ChatGPT, whereas ChatGPT supplies more relevant responses to queries. The results of ChatGPT may be out of date and often do not specify a validity time stamp. Google sometimes returns results based on commercial entities. The readability scores for both indicate that responses are often not appropriate for persons with low health literacy skills. In the future, the addition of both the source and the date of health-related information and availability in other languages may increase the value of these platforms for both nonmedical and medical professionals.

• Publisher: Gunther Eysenbach MD MPH, Associate Professor,JMIR Publications
• Subject: Alternative approaches / Artificial intelligence / Caregivers / Chatbots / Cognition / Cognitive Dysfunction / Cognitive impairment / Currency / Deep learning / Dementia / Health education / Health literacy / Human-computer interaction / Humans / Information / Interrater reliability / Keywords / Language / Language modeling / Languages / Literacy / Literacy skills / Machine learning / Medical personnel / Natural language / Objectivity / Original Paper / Queries / Readability / Reliability / Reproducibility of Results / Responses / Search Engine / Search engines / Secondary education / Semantics