Fully Convolutional Network and Region Proposal for Instance Identification with Egocentric Vision

This paper presents a novel approach for egocentric image retrieval and object detection. This approach uses fully convolutional networks (FCN) to obtain region proposals without the need for an additional component in the network and training. It is particularly suited for small datasets with low object variability. The proposed network can be trained end-to-end and produces an effective global descriptor as an image representation. Additionally, it can be built upon any type of CNN pre-trained for classification. Through multiple experiments on two egocentric image datasets taken from museum visits, we show that the descriptor obtained using our proposed network outperforms those from previous state-of-the-art approaches. It is also just as memory-efficient, making it adapted to mobile devices such as an augmented museum audio-guide.

1. Introduction

We propose to enhance a museum audio-tour guide with a camera, in order to help user orientation, enable automatic guidance and facilitate museum artifact explanations: when the visitor is close enough to an object, an explanation is automatically launched. The embarked camera is not used for augmented reality but only for the system to localize the user without the need of any extra hardware to be installed in the museum (a very strict constraint in some museums). Hence, only egocentric image analysis and object instance recognition is possible to localize the user on the museum map. The camera is installed on a small device held in the user's chest. Thus, this system needs to recognize object instances (not classes), such as paintings, sculptures, or any exhibited historical heritages. Obviously, the entire museum must be photographed (or video recorded), and each object image then has to be manually localized in the digital museum map.