A Crowd Source System for YouTube Big Data Analytics: Unpacking Values from Data Sprawl

YouTube has emerged as the most popular video platform across the world. This paper proposes a system for stream based meta-data analytics, to gain insights and uncover hidden patterns, for YouTube videos. The system reports the number of videos uploaded for each category, the videos with the highest views and highest likes etc. It crowd-sources the calls to the YouTube’s search and data APIs, and feeds it to Kafka Stream to process using PySpark. Finally, the processed video meta-data in stored in Apache Cassandra for downstream consumption. By experimenting with different time windows from 5 to 30 minutes, it is observed that 15-minute window is optimum for getting adequate video data de-duplication. From the available 30 video categories, with the minimum video length of 30 minutes, our study observed that the top 10 video categories with the highest number of videos are Film & Animation, Autos & Vehicles, Music, Sports, Travel & Events, Entertainment, News & politics, Documentary, Science & Technology and Education. The highest videos are uploaded under Entertainment category, which clearly captures the content creator’s interest.