Rare event detection by progressive clustering undersampling

Abstract

Capturing rare events in severely imbalanced datasets is challenging, as the learning and optimization processes are often biased toward the majority class. To address this issue, this study explores various resampling techniques and introduces a novel method called Progressive Clustering Undersampling (PCU). This technique removes negative instances that are distant from positive ones. PCU was compared with eight common undersampling and two oversampling techniques, consistently outperforming them on highly imbalanced and noisy datasets. The workflow demonstrates that rare anomalies can be effectively predicted using unsupervised methods based on frequency-driven decision boundaries. Progressive clustering ultimately identifies clusters with the highest concentration of positive instances. These delineated clusters are then saved by supervised models and used in the preparatory phase before prediction. The proposed method produces two outputs: one optimized for a high F1-score and the other for high precision. Overall, this approach presents a promising solution for identifying rare anomalies in complex, imbalanced data environments.