HCCD-DS v2: a transparent synthetic benchmark for human–AI decision support under contextual uncertainty

Abstract

Abstract Decision support pipelines increasingly combine machine learning predictions with human judgment, yet most public benchmarks evaluate model outputs only and do not encode the interaction process that determines final decisions. This limits reproducible analysis of when human intervention improves or degrades system-level performance. We introduce HCCD-DS v2, a transparent and configurable synthetic benchmark dataset that models decision-time interaction between an AI recommender and simulated users with continuous expertise levels under contextual uncertainty. Each decision instance links input context, system confidence, explainability metadata, human acceptance/override behavior, override rationale, and realized outcome. The dataset simulates distinct profiles for healthcare, cybersecurity, and Internet of Things (IoT) scenarios to support cross-domain evaluation. Empirical analyses and benchmark experiments show that human intervention is condition-dependent: expert overrides are more likely to improve outcomes, while novice overrides more often reduce success. By exposing this interaction in a unified and reusable resource under explicit behavioral assumptions (sequential trust updates, individual risk tolerance, and explanation sensitivity), HCCD-DS v2 provides a controlled environment for benchmarking trust calibration, explanation-aware interaction policies, and algorithms that learn to defer. We discuss the limits of predictive recoverability in simulated human behavior and frame the benchmark’s F1-score transitions as a measure of policy complexity.