A Comparative Analysis of the Accuracy of Retrieval Augmented Generation Versus Large Language Models for Gout-Related Queries

Authors

Samikchhya Keshary Bhandari, Tribhuvan University Teaching Hospital, Kathmandu, NepalFollow
Himal Kharel, Department of internal medicine, Rochester General Hospital, New York, USAFollow
Zeni Kharel, Department of hematology and oncology, Rochester General Hospital, New York, USAFollow
prajjwol bhatta, Department of internal medicine, Rochester General Hospital, New York, USAFollow
Mouna Reghis, Department of internal medicine, Rochester General Hospital, New York, USAFollow
Ali Mohamed, Department of internal medicine, Rochester General Hospital, New York, USAFollow
Eduardo Avalos Sugastti, Department of internal medicine, Rochester General Hospital, New York, USAFollow
Chengu Niu, Department of internal medicine, Rochester General Hospital, New York, USAFollow

Author Credentials

Samikchhya Keshary Bhandari¹, Himal Kharel², Zeni Kharel³, Prajjwol Bhatta², Mouna Reghis², Ali Mohamed², Eduardo Avalos Sugastti², Chengu Niu²

¹Tribhuvan University Teaching Hospital, Kathmandu, Nepal

²Department of internal medicine, Rochester General Hospital, New York, USA

³Department of hematology and oncology, Rochester General Hospital, New York, USA

Abstract

Importance: To better understand ways to improve the answers from large language models by feeding them disease specific guidelines which can potentially act as a clinical decision support tool in the future Objective: To demonstrate the effect of retrieval augmentation on answers related to gout in terms of accuracy, conciseness and unambiguity. Design: Observational. Setting: Virtual experimental setting. Participants: ChatGPT 3.5 and Retrieval augmented generation powered ChatGPT. Exposure: Nine questions derived from the 2020 American College of Rheumatology guidelines on gout. Main outcomes: A zero to two subjective scale to measure the accuracy, conciseness, and unambiguity of answers, objective measure for word count, and text similarity using Levenshtein distance. Results: There was a significant difference in the accuracy, conciseness, and unambiguity score between RAG and ChatGPT with the mean difference of 0.78; t(8)=2.4, p = 0.04, mean difference of 1.56; t(8)=8.85, p=0.00, and mean difference of 1;t(8)=3.46, p=0.00 respectively. In addition, the word count was also significantly lower in the RAG group with mean difference of 250.78; t(8)=23.99, p=0.00. The mean similarity score was 0.16(95% CI=0.06 to 0.26). Conclusion and relevance: These findings suggest that incorporating domain specific knowledge to LLM using RAG greatly enhance the accuracy, conciseness, and unambiguity of answers related to gout.

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License

Recommended Citation

Bhandari SK, Kharel H, Kharel Z, bhatta p, Reghis M, Mohamed A, Avalos Sugastti E, Niu C. A Comparative Analysis of the Accuracy of Retrieval Augmented Generation Versus Large Language Models for Gout-Related Queries. Advances in Clinical Medical Research and Healthcare Delivery. 2025; 5(3):60-62. doi: 10.53785/2769-2779.1288.