Data Ingestion - Cloud based Ingestion Analysis using NiFi

Data Ingestion has been an integral part of Data Analysis. Bringing the data from various heterogeneous sources to one common place and ensuring the data is captured in the appropriate format is the key for performing any Big data task. Data ingestion is performed using multiple frameworks across the industry and they all have their own set of benefits and drawbacks. Apache NiFi is one popular ingestion framework which is used widely and does Ingestion effectively. Ingestion is performed on various sources and the data is generally stored in clusters or cloud storage. In this paper, we have done the File Data Ingestion using the NiFi framework on a local machine and then on two cloud-based platforms, namely Google Cloud Platform (GCP) and Amazon Web Services (AWS). The objective is to understand the latency and performance of the NiFi tool on Cloud-based Ingestion and provide a comparative study against the typical Data Ingestion. The entire setup was done on a local machine and two corresponding cloud platforms namely GCP and AWS. The findings from the comparative analysis have been compiled in a tabular format and graphs are created for easy reference. The paper places emphasis on the significance of NiFi's data ingestion performance on Cloud Platform and attempts to present it as a major activity on the data ingestion platform for Cloud Ingestion Solution.