An Efficient Method of Data Encryption for Securing HDFS

Big data allows for the management of substantial quantities of different types of data i.e., structured, unstructured, and semi-structured. The business sector uses Big Data for a large range of tasks, such as enhancing customer service, managing real-time data flow between clients, and carrying out other successful operations. Big data offers businesses greater customer intelligence, better performance, more visibility into whole market performance, flexible supply chain management, data-driven innovation, smart recommendations, and targeting. A powerful system called Hadoop is primarily utilized for the distributed processing and processing of huge amounts of data. Since Hadoop does not provide any security, the encryption zone is used for security; nonetheless, this technique has a flaw from an attacker’s perspective. Encryption is a way to provide the security of large amounts of data traveling across the network or stored in a distributed system in a cloud environment. Hadoop provides different ways to encrypt the stored data. Disk encryption, application-level encryption, and HDFS data at REST are the encryption mechanisms developers use to ensure the security of big data sets. It takes a long time to encrypt large amounts of data that are stored in HDFS using the streaming approach. The data should be encrypted using a parallel process to minimize this time commitment. In this paper, the proposed encryption technique used to provide an efficient encoding scheme based on DES to provide better security of large data sets and also compare this encryption with other classical encryption techniques 3DES based on encryption time, decryption time, and total time taken by the mapper and also reducer. In this scheme, we integrated DES with compression technique and map-reduce framework. To secure HDFS, we use this novel approach, and certain experiment investigations are done to confirm its success.