On the Influence of Ill-conditioned Regression Matrix on Hyper-parameter Estimators for Kernel-based Regularization Methods

In this paper, we study the influence of ill-conditioned regression matrix on two hyper-parameter estimation methods for the kernel-based regularization method: the empirical Bayes (EB) and the Stein's unbiased risk estimator (SURE). First, we consider the convergence rate of the cost functions of EB and SURE, and we find that they have the same convergence rate but the influence of the ill-conditioned regression matrix on the scale factor are different: for upper bounds, the scale factor for SURE contains one more factor cond(Φ
^T
Φ) than that of EB, where Φ is the regression matrix and cond(·) denotes the condition number of a matrix. This finding indicates that when Φ is ill-conditioned, i.e., cond(Φ
^T
Φ) is large, the cost function of SURE converges slower than that of EB. Then we consider the convergence rate of the optimal hyper-parameters of EB and SURE, and we find that they are both asymptotically normally distributed and have the same convergence rate, but the influence of the ill-conditioned regression matrix on the scale factor are different. In particular, for the ridge regression case, we show that the optimal hyperparameter of SURE converges slower than that of EB with a factor of 1/n
²
, as cond(Φ
^T
Φ) goes to ∞, where n is the FIR model order.