A Compendious Research on Big Data File Formats

In modern times, the net amount of data generated is significantly higher than in previous times, and such data is enormous, complex and hard to manage. The field which deals with such kind of data is known as Big Data. Big Data has helped people understand and get a much better grasp of such complex data. It has also helped extract and analyse information from such data. Big Data is a prominent and promising field, and the three essential factors of big data are volume, variety, and velocity. Big data can be stored in different file formats. File formats play a vital role in big data, influencing the two most essential factors, volume and velocity. The file formats have to be chosen very carefully as they have a domino effect over the speed or velocity of the particular tasks that have to be carried out on the big data. So, a comprehensive study of the following formats CSV (Comma-Separated Values), JSON (JavaScript Object Notation), Avro, Parquet, ORC (Optimized Row Columnar) and XML (Extensible Markup Language), has been carried out. The quantitative analysis has been carried both on premise and on the cloud.