Stochastic three-term conjugate gradient: a third-order curvature approximation correction algorithmic framework for machine learning

Abstract

Abstract Classical conjugate gradient methods rely solely on first-order information, which limits their ability to capture curvature information in nonconvex stochastic optimization. To address this limitation, this paper proposes a stochastic three-term conjugate gradient algorithm incorporating third-order curvature approximation (TASCG). By integrating third-order tensor information into the search direction, the proposed algorithm enhances its ability to capture the local geometry of nonconvex loss landscapes, while satisfying both the sufficient descent property and boundedness conditions without additional assumptions. Under standard assumptions of gradient Lipschitz continuity and bounded variance, we rigorously establish global convergence of the TASCG algorithm and derive a stochastic first-order oracle complexity bound of $$\mathcal {O}(\epsilon ^{-2})$$ , which matches the optimal complexity of classical stochastic gradient methods. To further improve gradient estimation accuracy, we incorporate variance reduction techniques into the TASCG framework, resulting in the variant TASCG-VR. Numerical experiments on nonconvex SVM and empirical risk minimization problems demonstrate that the proposed algorithms significantly outperform standard SGD and SVRG in terms of convergence speed and final solution accuracy, while being more robust to step-size selection. This work provides a novel theoretical perspective and algorithmic framework for integrating higher-order geometric information into stochastic conjugate gradient methods.