LASSO: A feature selection technique in predictive modeling for machine learning

Feature selection is one of the techniques in machine learning for selecting a subset of relevant features namely variables for the construction of models. The feature selection technique aims at removing the redundant or irrelevant features or features which are strongly correlated in the data without much loss of information. It is broadly used for making the model much easier to interpret and increase generalization by reducing the variance. Regression analysis plays a vital role in statistical modeling and in turn for performing machine learning tasks. The traditional procedures such as Ordinary Least Squares (OLS) regression, Stepwise regression and partial least squares regression are very sensitive to random errors. Many alternatives have been established in the literature during the past few decades such as Ridge regression and LASSO and its variants. This paper explores the features of the popular regression methods, OLS regression, ridge regression and the LASSO regression. The performance of these procedures has been studied in terms of model fitting and prediction accuracy using real data and simulated environment with the help of R package.

I. Introduction

Statistics is a field that faces the challenges and problems from the areas of science and industry. These challenges and problems have exploded due to the invention and rapid growth of computers and information technologies. As the amount of data increased predominantly, extracting the patterns and trends, and understanding the data (i.e., learning from data) is required. This led to a new field known as “machine learning”. The amount of data is increasing each and every second and making sense of this enormous amount of data is a challenging task for data-driven industry people. The data is composed of a number of features. It mainly focuses on supervised and unsupervised learning. Supervised learning postulates the machine to learn from the data when a target variable is specified. Thus, the machine's task is reduced to only divine some pattern to get the target variable from the input data. Unsupervised learning is just the contrary of supervised learning. In which, the target value or label is not mentioned for the data. It also visualizes the data in two or three dimensions.