A Hybrid Approach to Data Pre-processing Methods

This is an era of big data, as data is growing exponentially and resources are running out of infrastructure, so it is required to accommodate all the data that gets generated. We collect data in enormous amounts to derive meaningful conclusions, perform effective data analytics and improve decision making. As we don't have enough infrastructures to support data storage for huge volumes, it is needed to clean the data in compulsion. It is a mandatory to carry out a step before doing anything with the data. We call it pre-processing of data and this is carried out in various steps. Pre-processing includes data cleaning, data integration, data filtering, and data transformation and so on. As such preprocessing is not limited to the number of steps or a number of methods or definitive methods. We must innovatively preprocess the data before it is being consumed for data analytics. It has become a responsibility for every data analyst or big data researcher to handpick data for his or her analytics. Considering all these techniques in mind we are proposing a hybrid technique to leverage various algorithms available to pre-process our data along with minor modifications such as at the run time, choosing an algorithm or technique wisely based on the data that we have.