A Flexible Hybrid Quantum-classical Training Framework of Organ-at-Risk and Tumor Segmentation Models for Radiation Therapy Planning

Abstract

Abstract Deep learning-based Organ-at-Risk (OAR) and tumor segmentation is vital for radiation therapy planning but often suffers from over-parameterization, requiring large datasets to avoid overfitting, which is impractical in small-sample medical settings. Traditional trainable parameter reduction methods, relying on structural lightweighting or low-rank approximation, may artificially limit model expressiveness and hurt performance. We propose a Hybrid Quantum-Classical Training Framework (HQC-TF) based on the Quantum Parameter Generation (QPG) technique to reduce trainable parameters while preserving model structure and adaptively determining parameter matrices’ ranks during training. This retains representational flexibility with parameter efficiency. HQC-TF uses independent Variational Quantum Circuits (VQCs) per channel, preserving channel independence and applying flexibly to deep neural network training. Experiments showed it significantly improved segmentation with fewer parameters compared to the classical training framework: UNetPP gained 6.77% IoU and 3.09% DSC for kidney tumors. Notably, it operates only during training via shallow quantum circuits, making it a practical, scalable solution for near-term clinical use in radiation therapy.