Sparse Weighted Naive Bayes Classifier for Efficient Classification of Categorical Data

Feature selection has become a key challenge in machine learning with the rapid growth of data size in real-world applications. However, existing feature selection methods mainly focus on numeric data, which will lead to quality loss when handling classification problems involving categorical variables. In this paper, we proposed an improvement of Bayesian Classifier with the sparse regression technology. To the best of our knowledge, this is the first attempt to extend sparse regression for directly process of categorical variables. We implemented the idea for the case of weighted naive Bayes classifier. The introduction of L1 regularized learning ensures the algorithm to retain only a minimal subset of variables for model building, while at the same time achieves a near-optimal decision hyper-plane, which leads to excellent performance in case of high dimensional or small sample size situations. We carried out benchmark test on five UCI benchmark categorical data sets, which proved that the proposed algorithm have competitive performances over the original weighted naive bayes classifier and several state-of-the-art feature selection methods including L1 logistic regression and SVM-RFE.