Generalized Algorithm for Website Parsing

In this paper, the process of parsing any website is formalized and represented as a generalized algorithm. For this purpose, the notion of target elements containing or leading to information to be downloaded is introduced. Such information can be either a certain text fragment on a web page or a file to be downloaded. Multilevel nesting of elements through element list processing loops is taken into account. It also takes into account the transition to the next pages (pagination) and the reloading of the web page to get the full list of items. A short review of papers that have investigated similar problems is given, but no attempt has been made there to fully generalize the parsing process.