Design and Implementation of a Crawler-Based Bad Web Page Discovery System

With the advent of the era of big data and 5G., people's demand for browsing the web is increasing., and the quality of web content is crucial to the user's browsing experience., but some web sites still have bad web content., such as pornography., violence., gambling., etc., which brings security risks to social security. In order to effectively identify and filter bad web content., this paper proposes a crawler-based bad web page discovery system. This paper first introduces the functional design., process design., neural network design and other aspects of the bad web page detection system., and elaborates on the implementation of the core functions. The final test results show that the crawler-based bad web page discovery system can effectively improve the discovery and filtering ability of bad web pages and meet the actual needs of current users.