A Third-Order Majorization Algorithm for Logistic Regression With Convergence Rate Guarantees

In this paper, we study the classical Logistic Regression (LR) problem in machine learning. Traditionally, the solving algorithms are based on either the first- or second-order approximation of the objective. For instance, the Fixed-Hessian Newton (FHN) method approximates the true Hessian with a constant estimate. In contrast, our design additionally exploits the third-order information. Applying the majorization–minimization (MM) framework, we construct a novel majorizing function based on the third-order Taylor expansion and the minimization solution is in closed-form with perseverance of the true gradient and Hessian structures. In analysis, we prove the convergence rate of the proposed algorithm. The enhanced numerical performance can be verified through simulation results.