Soft Upper-bound Support Vector Machines

The minimum complexity L1 SVM (ML1 SVM) is the fusion of the minimal complexity machine (MCM) and the standard support vector machine (L1 SVM) to realize high generalization ability over the L1 SVM. The fusion is realized by minimizing the upper bound on the decision function values (i.e., minimizing the maximum margin as well as maximizing the minimum margin) in the L1 SVM. In this paper, to avoid worsening the generalization ability by minimizing the upper bound, we introduce the soft upper bound on the decision function values. This is realized by allowing some of the training data outputs to exceed the upper bound. We call this architecture soft upper-bound L1 SVM (SL1 SVM). By introducing the soft upper-bound, the number of hyperparameters increases by one. We explain how to cancel the increase by setting the hyperparameter value properly, and clarify the characteristics of the SL1 SVM. Finally, we compare the SL1 SVM with ML1 SVMs using several benchmark data sets and show that overfitting occurs for the SL1 SVM more frequently than for the ML1 SVMs.