Dynamic Analysis Pipeline in Real Time Monitoring Scenario

Social networks offer researchers a unique opportunity to study human communication and social phenomena on an unprecedented scale. However, the variability in the amount of data generated at different times poses a challenge to research in this area. This paper presents an analysis pipeline that uses Twitter data to efficiently process and analyze social media posts in real time. It does not require resources to be designed for maximum load. The approach is based on three key components: implementing analysis processes as independent microservices, using a publish-subscribe communication system, and implementing automatic scaling mechanisms. These factors enable a flexible, robust, and scalable analysis pipeline. It can dynamically adapt to changing workloads and ensure optimal resource utilization.