Recommending Restaurants based on Classification of Photos from the Gallery of Mobile Device

The restaurant recommender systems are important for travelers, but may suffer from the “cold start” problem for new users. In this paper, it is assumed that photos of food in a gallery of mobile device of a user reflect his or her gastronomic preferences. The novel pipeline for modeling of the user's food preferences is introduced. At first, scene recognition neural network is used to extract photos of the restaurants from a gallery. It is proposed to train the multi-task convolutional network to predict the presence of food in an input image and the attributes of the restaurant. The recognized types of cuisine are summarized in a profile of gastronomic preferences of the user. The restaurants in a given city are recommended based on these preferences and additional attributes of a restaurant such as cumulative rating. Experimental study for the Yelp datasets is provided. Several neural network architectures are compared, among which EfficientNet models demonstrated the best performance.