Retinal lesion annotation on fundus imaging: an interobserver variability study

Abstract

Abstract Accurate detection and segmentation of retinal lesions in fundus photographs is essential for diagnosing diabetic retinopathy (DR) and for developing reliable machine learning models for this purpose. However, the reliability of manual annotation of retinal lesions by clinical experts has yet to be evaluated. We quantified interobserver variability in the pixel-wise annotation of microaneurysms (µA), intraretinal hemorrhages (Hem), hard exudates (Ex), and cotton-wool spots (CWS) using 51 images from the MAPLES-DR dataset, independently annotated by three senior retinologists. Manual segmentation showed substantial variability: 38% of lesions marked by one observer were undetected by the others, lesion coordinates differed by approximately one third of the lesion diameter, and the mean interobserver IoU was 0.47, indicating inconsistent contour delineation even when observers detected the same lesion. Paradoxically, inter-expert DR grading assessments were largely concordant, suggesting reliance on more holistic or experience-based judgment criteria rather than on lesion counting. We assessed a semi-automated workflow in which annotators corrected model-generated pre-segmentations. This approach markedly improved agreement, yielding a pixel-wise mean IoU of 0.71, reducing both detection and contour-related variability; over 90% of lesions were preserved directly from the pre-segmentation.