Big Data Streaming Architecture for Edge Computing Using Kafka and Rockset

Due to the ever-increasing growth of continuously generated data streaming applications, there is a need for adopting new streaming data architectures to support low-latency between the source and destination. This work proposes an architecture to extract and pipeline Twitter’s streaming data using Kafka, complemented by Rockset, and visualize the analytics with Tableau. We developed an application to capture tweets for each trend and ingesting the data into Kafka. The data aggregation and processing occurred in Kafka before storing it in the cloud using Kafka producers and consumers. The only consumer we used is Rockset, a real-time streaming analytics tool to filter the data by writing queries. Finally, the data Rockset is connected to Tableau to visualize real-time trends using treemaps on dashboards and storyboards.