Objective: To investigate the performance of three artificial intelligence (AI)-powered large language models (LLMs) as adjunctive tools in analyzing clinical pictures of common phoniatric disorders. Methods: Medical history, symptoms, and videolaryngostroboscopic images of adult outpatients with laryngological disorders consulting in laryngology were presented to ChatGPT-4o, DeepSeek, and Claude-3.7-Sonnet for primary and differential diagnoses, management, and treatment. The accuracy of LLMs was assessed by two blinded laryngologists using the artificial intelligence performance instrument (AIPI). Errors of LLMs were documented. Results: Thirty-nine patients completed the evaluations [19 (48.7%) females]. The mean age was 55.4 ± 19.6 years. Of the 52 identified phoniatric disorders, glottic insufficiency (n = 8, 20.5%), laryngopharyngeal reflux disease (n = 7, 17.9%), Reinke edema (n = 6, 15.4%), and vocal fold nodules (n = 5, 12.8%) were the most prevalent conditions. The performances of LLMs were low, with correct primary and differential diagnoses ranging from 15.4% to 28.2%, and 15.4% to 25.6%, respectively. AIPI scores were comparable across LLMs, except for treatment recommendations, with DeepSeek demonstrating significantly lower scores than Claude-3.7 and ChatGPT-4o (P = 0.019). The judges reported a high intraclass consistency (ICC) in the AIPI assessment [ICC = 0.888; 95% confident interval (0.831–0.932)]. The conditions associated with the highest inaccurate diagnosis across LLMs were reflux (n = 21), glottic insufficiency (n = 20), Reinke edema (n = 19), and vocal fold cyst (n = 16). ChatGPT-4o reported more accurate scores for difficult clinical cases (rs= 0.410; P = 0.008). Conclusion: The performance of AI-powered LLMs is mild for providing accurate analysis of phoniatric cases based on clinical findings and laryngostroboscopic images. Future studies are needed when updated LLM versions will consider video analyses rather than only clinical images.
Accuracy of AI-Powered Large Language Models in the Analysis of Laryngeal Images
Maniaci A.;
2025-01-01
Abstract
Objective: To investigate the performance of three artificial intelligence (AI)-powered large language models (LLMs) as adjunctive tools in analyzing clinical pictures of common phoniatric disorders. Methods: Medical history, symptoms, and videolaryngostroboscopic images of adult outpatients with laryngological disorders consulting in laryngology were presented to ChatGPT-4o, DeepSeek, and Claude-3.7-Sonnet for primary and differential diagnoses, management, and treatment. The accuracy of LLMs was assessed by two blinded laryngologists using the artificial intelligence performance instrument (AIPI). Errors of LLMs were documented. Results: Thirty-nine patients completed the evaluations [19 (48.7%) females]. The mean age was 55.4 ± 19.6 years. Of the 52 identified phoniatric disorders, glottic insufficiency (n = 8, 20.5%), laryngopharyngeal reflux disease (n = 7, 17.9%), Reinke edema (n = 6, 15.4%), and vocal fold nodules (n = 5, 12.8%) were the most prevalent conditions. The performances of LLMs were low, with correct primary and differential diagnoses ranging from 15.4% to 28.2%, and 15.4% to 25.6%, respectively. AIPI scores were comparable across LLMs, except for treatment recommendations, with DeepSeek demonstrating significantly lower scores than Claude-3.7 and ChatGPT-4o (P = 0.019). The judges reported a high intraclass consistency (ICC) in the AIPI assessment [ICC = 0.888; 95% confident interval (0.831–0.932)]. The conditions associated with the highest inaccurate diagnosis across LLMs were reflux (n = 21), glottic insufficiency (n = 20), Reinke edema (n = 19), and vocal fold cyst (n = 16). ChatGPT-4o reported more accurate scores for difficult clinical cases (rs= 0.410; P = 0.008). Conclusion: The performance of AI-powered LLMs is mild for providing accurate analysis of phoniatric cases based on clinical findings and laryngostroboscopic images. Future studies are needed when updated LLM versions will consider video analyses rather than only clinical images.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
