OR02 - Assessing the Validity of Chatbot Responses in Endodontic Diagnosis and Treatment Plans with Clinical Vignettes

Wednesday, April 2, 2025

10:15 AM – 10:30 AM EST

Location: Hynes Convention Center, Floor 3, Room 309

Presenter(s)

SM

Seon Young Min, D.D.S.

Endodontic resident
Oregon Health & Science University
San Francisco, California

Disclosure information not submitted.

Co-Author(s)

Karan J. Replogle, D.D.S., M.S.

Associate Professor of Endodontics, School of Dentistry Senior Associate Dean, Clinical Systems
Oregon Health & Science University
Portland, Oregon

Disclosure information not submitted.
DC

Dongseok Choi, Ph.D.

Professor
Oregon Health & Science University
Portland, Oregon

Disclosure information not submitted.
AL

Adam Lloyd, B.D.S., M.S.

Professor, Division Head of Endodontics & Advanced Education Program Director
Oregon Health & Science University
Portland, Oregon

Disclosure information not submitted.
Eun Park, D.D.S., Ph.D.

Assistant professor of Endodontics
Oregon Health & Science University
Portland, Oregon

Disclosure information not submitted.

Objectives: To assess the accuracy and comprehensiveness of chatbot-generated responses for endodontic diagnoses and treatment plans using clinical vignettes.

Methods: GPT-4o, Copilot, Gemini, and Gemini Advanced provided endodontic diagnosis and recommended treatment plans for 7 clinical vignettes from the AAE Colleagues for Excellence (2013). Data were collected twice, 10 days apart, resulting in 56 responses. Two faculty endodontists independently evaluated the responses, resolving scoring disagreements through evidence-based discussion. Responses were graded based on: (1) binary accuracy of the pulpal and periapical diagnosis (correct/incorrect) and (2) descriptive accuracy (6-point Likert scale, with 1 being completely incorrect and 6 being completely correct) and completeness (3-point Likert scale, with 1 being incomplete and 3 being complete with additional context). Scores were analyzed using mixed-effects models. The study, including data analysis, was conducted from June Oct 2024.

Results: Overall binary accuracy for pulpal diagnosis was 87.5%, with no significant differences among the chatbots (p=0.100). For periapical diagnosis, accuracy was 76.7%, with a significant difference (p=0.026): Gemini Advanced at 100%, ChatGPT-4o at 85.7%, Copilot at 64.3%, and Gemini at 57.1%. Descriptive accuracy differed significantly among chatbots (p=0.001), with Gemini Advanced scoring 6 on the 6-point Likert scale in 92.9% of responses, followed by ChatGPT-4o at 64.3%, and both Copilot and Gemini at 42.9%.

Conclusion: Chatbots demonstrated high accuracy in pulpal diagnosis; however, their performance in periapical diagnosis and descriptive accuracy varied significantly across models, with Gemini Advanced achieving the highest validity.