Assessment of the proficiency of ChatGPT-4o in an image-based robot-assisted radical prostatectomy scenario.

Abstract

423 Background: Surgery is a process that heavily relies on complex visual information. Many AI-based surgical teaching and assistance applications have emerged. However, the application of the newly released ChatGPT-4o in surgical contexts remains unexplored. This study aims to investigate the ability of ChatGPT-4o in analyzing and recognizing anatomical structures, surgical procedures, and operational precautions based on image information in a surgical context. Utilizing screenshots from robot-assisted radical prostatectomy videos sourced from the internet. Methods: We developed a test comprising 77 images and 65 questions, with a total score of 100 points. The test includes sections on anatomical structures, surgical procedures, and operational precautions. We sequentially input the questions into the ChatGPT-4o model and employed strategies to enhance its performance during the test. A passing score was set at 60 points or above, and an excellent score at 80 points or above. Results: ChatGPT-4o achieved a score of 68, surpassing the passing threshold. The accuracy for non-image questions was higher than for image-based questions (85.71% vs. 58.46%, χ² = 7.78, p = 0.005), and questions with fewer images had a higher accuracy than those with multiple images (71.11% vs. 30.00%, p < 0.001). Furthermore, we found ChatGPT-4o performed differently in recognizing different structures, with the highest accuracy for questions involving muscle and fascia (100%) and the lowest for those involving nerve and vessel structures (33.33%). Conclusions: ChatGPT demonstrated commendable abilities in image recognition and problem analysis in surgical contexts. We hope that the results of this study will contribute to future applications of ChatGPT in AI-assisted surgical teaching and assistance.