A Framework for the Efficient Collection of Big Data from Online Social Networks

In this paper, we present a universal framework for collecting publicly available information from Online Social Networks (OSNs). Our proposal is based on a three-levels distributed architecture. At a first level, one or more crawler parallel processes are in charge of identifying all the resources that need to be acquired from a target OSN. Once identified, these resources are requested to the intermediate-level. This level implements an abstraction layer that allows crawlers to query at the same time different OSNs using the same interface. Here, if the needed resource is available, it is returned immediately. Otherwise, a new request is prepared and shared toward a network of remote data collectors processes by means of a set of distributed data structures. The architecture is organized to allow a large number of data collectors to operate in parallel, so to make it possible to download big amount of data in a relatively short amount of time. In our paper, we also present the results of some experiments we conducted on the Twitter and the Flickr OSNs to validate our framework.