OR25 - Large Language Model as A Virtual Endodontic Patient. Can They Help Us Teach Endodontic Diagnosis?

Objective: In dental education, actors are widely used as standardized patients to train students in communication and diagnostic skills. Large language models (LLM) can be adapted to function as virtual patients. This project compares five LLMs to determine the most accurate answers and persona (character maintenance).

Methods: Calibration of five LLMs (GPT-4.0, Google Gemini, Google Gemini NYU, GPT-4.0 NYU, and Llama) was performed using 10 different endodontic cases. LLMs were prompted with 18 consistent benchmarking questions across all cases. After data collection, the average performance score was calculated for each LLM across all prompts. Two scores were calculated: Score 1 measured the correctness of answers; and Score 2 measured persona. A point system of 1-5 was used to quantify the LLMs’ performance for each score, with 10 points possible per interaction. Higher scores indicated better performance. LLM scores were compared using the least-squares means using the Tukey-Kramer test to assess statistically significant differences (p < 0.05).

Results: GPT-4.0 NYU provided the highest average scores (Score 1, 4.45; Score 2, 4.42). Llama provided the lowest average scores (Score 1, 3.51; Score 2, 3.39). Score 1 for GPT-4.0 NYU, GPT-4.0, and Google Gemini NYU had no statistically significant difference. For Score 2, GPT-4.0 NYU performed best (p < 0.05).

Conclusion: GPT-4.0 NYU outperformed other LLMs in both scores, therefore the most suitable option for use as a virtual patient in dental education. The accessibility of LLMs make them a useful, low-cost learning tool for application in dental education.