Distributed Collection Method of Economic Growth Data Based on Cloud Computing

First of all, according to the actual data collection scene, combined with the advantages and disadvantages of the existing detector framework, the overall architecture of the distributed data collection system is proposed, and it is divided into three parts: web management side, server side, collection side. Secondly, on the basis of the above design, the development language is based on Python, and then the acquisition system is developed on the cloud platform in the laboratory, and a detailed test is carried out for the real field. Finally, use the website as a source for collecting data. This website contains two types of anti-tracking measures: IP Restriction and Referral Restriction. The number of pages successfully collected is counted every 10 minutes. The experimental results show that the number of pages that are not successfully requested by the download middleware does not increase since the acquisition system starts from 20 minutes. For the collection node that triggers the download of the middleware, the dynamic IP proxy setting is enabled, so even if the collection can be resumed after 30 minutes, accessing the data source through the proxy server will undoubtedly increase the network delay. Therefore, downloading middleware has a significant impact on setting reference restrictions, IP blocking, etc.