Object detection in histology: A multi-dataset benchmark and test-time inference

Abstract

Medical image analysis has become increasingly important for automated medical diagnosis, as well as deep learning. Specifically, object detection models may help in automatically identifying pathological structures and features. This study presents a comprehensive comparative analysis for object detection tasks in histological images of the latest models including the YOLO (You Only Look Once) architectures, from YOLOv8 to the recently introduced YOLOv12. These models were evaluated alongside alternative architectures including RT-DETR, YOLO-World, and YOLOE across five diverse histology datasets: BCNB, Nuclei, TNBC, MoNuSAC, and CryoNuSeg. The experimental analysis employed standardized training protocols with consistent hyperparameters and data augmentation strategies, evaluating the performance through multiple metrics, inference time, and computational cost. The results obtained on the five datasets indicate that YOLOv11 consistently showed a strong performance across multiple datasets, however the newly introduced attention mechanisms of YOLOv12 show good performance, despite the model having slightly lower overall performance. Specialized variants like YOLOE demonstrated promising results for specific applications, while RT-DETR showed poor performance on smaller objects, which are typical in histological images. Statistical analyses indicate that YOLOv11 indeed has the best performance but that all models have a poor performance on objects of small sizes; moreover, the most common cases of failure are background false positives and missed detections. This comprehensive evaluation provides insights for the current state of object detection architectures for clinical histopathology applications and establishes benchmarks for future avenues of research in automated medical image analysis. In addition to the multi-model benchmark, we propose Test-time Graph Similarity Propagation (TGSP), a test-time self-supervised refinement that uses ResNet50 deep features to build a k-NN similarity graph over detections and performs label propagation to re-score predicted boxes. TGSP replaces TSBP’s iterative Earth-Mover matching with adaptive per-class quantile thresholds and graph-based label propagation, eliminating K-means hyperparameters and better scalability. Our analysis on histology datasets TGSP consistently matches or improves F1 relative to both a fixed 0.5 threshold and TSBP, with the biggest gains when base-model confidence calibration is poor.