Cyber vulnerabilities surveillance via web-scraped specialized documents

Cyber security articles represent an important source of information of up-to-date data about software vulnerabilities, including data like Indicators of Compromise, highly valuable to any security researcher. The articles can be collected using Crawlers and Scrapers, and can be later analyzed in order to create keyword-type searches. This paper conducts a study towards the extraction of articles from forum-like websites, and of relevant information from the articles, while highlighting different types of Crawlers, Scrapers and analysis algorithms, such as the extraction of Indicators of Compromise, keyword text search and Natural Language Processing methods. Thus, we will present the types of forum websites with their characteristics, alongside different types of Crawlers, while also presenting some of the benefits of ethical scraping and some of its most important principles, such as postponing a request to server and changing the User-Agent header. The extracted information can be saved in an Elasticsearch Index, in order to create various statistics about the information extracted, providing an up-to-date overview over the current state of the cybersecurity space.