ChatGPT as an information tool in rhinology. Can we trust each other today?

Abstract

Purpose: ChatGPT (Chat-Generative Pre-trained Transformer) has proven to be a powerful information tool on various topics, including healthcare. This system is based on information obtained on the Internet, but this information is not always reliable. Currently, few studies analyze the validity of these responses in rhinology. Our work aims to assess the quality and reliability of the information provided by AI regarding the main rhinological pathologies. Methods: We asked to the default ChatGPT version (GPT-3.5) 65 questions about the most prevalent pathologies in rhinology. The focus was learning about the causes, risk factors, treatments, prognosis, and outcomes. We use the Discern questionnaire and a hexagonal radar schema to evaluate the quality of the information. We use Fleiss's kappa statistical analysis to determine the consistency of agreement between diferent observers. Results The overall evaluation of the Discern questionnaire resulted in a score of 4.05 (±0.6). The results in the Reliability section are worse, with an average score of 3.18. (±1.77). This score is afected by the responses to questions about the source of the information provided. The average score for the Quality section was 3.59 (±1.18). Fleiss's Kappa shows substantial agreement, with a K of 0.69 (p<0.001). Conclusión: The ChatGPT answers are accurate and reliable. It generates a simple and understandable description of the pathology for the patient's beneft. Our team considers that ChatGPT could be a useful tool to provide information under prior supervision by a health professional.