Calibrated early-warning models with fairness auditing and selective prediction for course withdrawal risk: Evidence from OULAD

Abstract

Early-warning systems (EWS) in learning analytics are increasingly used to identify learners at risk of course withdrawal, but their deployment-critical properties are often under-reported once predicted scores are converted into intervention policies. This study develops a deployment-oriented evaluation protocol for course-withdrawal risk using the Open University Learning Analytics Dataset (OULAD). An early-window feature set was constructed from the first four weeks of learner activity and evaluated under a group-wise train–test split by course presentation. Multiple classifiers were benchmarked, including logistic regression, histogram-based gradient boosting (HGB), random forest, support vector machine, AdaBoost, K-nearest neighbors, XGBoost, LightGBM, and CatBoost. A calibrated HGB model was then retained as the main probabilistic model for downstream analyses of probability reliability, threshold sensitivity, subgroup fairness with bootstrap uncertainty, selective prediction, and capacity-based Top- x % alerting. Several tree-based and boosting models achieved comparable held-out discrimination, while calibrated HGB remained competitive across classification, ranking, and probability-reliability metrics. Threshold choices substantially changed the precision–recall balance, indicating that operating points should be treated as policy choices rather than universal defaults. Fairness audits showed policy-dependent observed group-level differences, especially in alert rates for disability status, although several subgroup error-rate and positive predictive value (PPV) differences remained uncertain. Selective prediction reduced risk on accepted cases as coverage decreased, whereas Top- x % alerting fixed outreach volume and made workload–effectiveness trade-offs explicit. Robustness analyses supported the 28-day window as a practical early-warning compromise and showed that absolute PPV values varied across held-out course-presentation splits. The findings suggest that EWS should be evaluated as policy-linked decision systems, integrating model benchmarking, calibration, fairness uncertainty, and capacity-aware decision rules before deployment.