Comparison of sampling methods in machine learning models

Abstract

Abstract This study evaluates the performance of seven sampling strategies, including five probability-based sampling methods–Simple Random Sampling (SRS), Stratified Sampling (SS), Importance Sampling (IS), Maximum Ranked Set Sampling with Unequal Samples (MRSSU), and the proposed Stratified MRSSU (SM)–together with the synthetic oversampling methods Synthetic Minority Over-sampling Technique (SMOTE) and Adaptive Synthetic Sampling (ADASYN), in conjunction with four machine learning models: Generalized Linear Model (GLM), Random Forest (RF), Support Vector Machine (SVM), and Extreme Gradient Boosting (XGB). The proposed SM method combines stratified sampling with MRSSU and employs a normalized auxiliary ranking score together with class-specific ranking directions to provide a structured ranking mechanism for selecting observations under class imbalance. A comprehensive Monte Carlo simulation study is conducted under different class imbalance ratios and sample sizes. The proposed approach is further evaluated on two real-world benchmark datasets, namely the Pima Indians Diabetes and Wisconsin Breast Cancer datasets, and compared with conventional sampling strategies as well as synthetic oversampling methods. Performance is assessed using Accuracy, Precision, Recall, F1-score, and the Area Under the Receiver Operating Characteristic Curve (AUC), while Friedman and Nemenyi tests together with mean-rank analysis are used for statistical comparison. The results indicate that the proposed SM method provides competitive performance across a range of simulation scenarios and real-world datasets, particularly under small sample sizes and class imbalance, although no single sampling strategy consistently outperformed all competitors. These findings suggest that combining stratification with class-specific ranked-set sampling can provide a competitive alternative for classification problems involving class imbalance.