Web news analysis based on web crawler technology

Since the concept of big data was put forward, Internet data has received wide attention and data mining objects, and network news data occupies half of the Internet data. With the characteristics of rapid dissemination, short exposure time and containing netizens’ opinions, the intrinsic value contained in web news is self-evident. In this paper, we use relevant web crawler technologies and algorithms to build an automated collection of online media news data and structured storage. The Chinese word separation algorithm and Chinese similarity analysis algorithm are used to summarize and organize the system, and the system is tested to check the validity and accuracy of the system design. According to the test results, the system can collect and organize information more accurately, so the system results are valid and can be used as a tool for analyzing online news in the future.