Contents I. INTRODUCTION
Every day, modern cities generate massive and complex, heterogeneous and diverse, large-scale spatio-temporal data, which contains rich value information. However, these data are distributed on Internet pages dispersedly, posing new and higher requirements for information collection and processing methods. Most of the traditional information collection and processing methods collect limited and regular information data, and cannot batch acquire, store and manage massive information. As a new thing in the Internet era, big data has gradually become a new idea and tool for human beings to understand the inherent laws of objective things, and has broad application prospects in information processing, technology research and development, commercial services, and medical diagnosis [1]. As a kind of big data technology, in recent years, some scholars at home and abroad have tried to use web crawler technology to obtain massive information on the Internet. They have applied the technology to information acquisition and analysis, and used data mining technology to manage, integrate and mine large-scale data [2]. From the current research results, web crawler technology has a good application prospect for all kinds of information acquisition, and will become one of the important methods of information acquisition research [3].
