Adaptive federated clustering for uncertainty-aware learning on decentralized big data platforms

Abstract

Federated learning (FL) struggles with scalability in decentralized big data platforms due to data heterogeneity, communication bottlenecks, and computational inefficiencies. We propose Adaptive Federated Clustering (AFC), a novel framework that addresses these challenges through three key innovations: (1) adaptive client selection based on computational capacity and data relevance, (2) hierarchical aggregation organizing clients into clusters for localized updates, and (3) sparsity- and quantization-based model compression. Experiments on CIFAR-10, CIFAR-100, Fashion-MNIST, and MIMIC-III demonstrate AFC achieves 4.3% higher accuracy than FedAvg, 49% lower communication cost, and 35% faster convergence. Under backdoor attacks, AFC shows only 2.8% accuracy degradation versus 7% for FedAvg. While AFC assumes relatively stable network connectivity and does not yet support fine-grained personalization, it significantly outperforms existing algorithms in scalability, robustness, and efficiency. These results demonstrate AFC’s practical value for secure collaborative learning on decentralized platforms, particularly in healthcare and IoT applications where bandwidth constraints and data heterogeneity are prevalent.