Joint Object and State Recognition Using Language Knowledge

The state of an object is an important piece of knowledge in robotics applications. States and objects are intertwined together, meaning that object information can help recognize the state of an image and vice versa. This paper addresses the state identification problem in cooking related images and uses state and object predictions together to improve the classification accuracy of objects and their states from a single image. The pipeline presented in this paper includes a CNN with a double classification layer and the Concept-Net language knowledge graph on top. The language knowledge creates a semantic likelihood between objects and states. The resulting object and state confidences from the deep architecture are used together with object and state relatedness estimates from a language knowledge graph to produce marginal probabilities for objects and states. The marginal probabilities and confidences of objects (or states) are fused together to improve the final object (or state) classification results. Experiments on a dataset of cooking objects show that using a language knowledge graph on top of a deep neural network effectively enhances object and state classification.

1. INTRODUCTION

Image classification is a research area in computer vision that has gained great attention in recent years mainly to tackle object classification and detection problems [1], [2], [3]. Object states, on the contrary, have not been considered as much as object classification in recent literature. Moreover, object states require further analysis especially for robotics-based applications. Robotic manipulation, task planning, and grasping require knowledge and constant feedback about the state of the environment and objects. For instance, if a robot chef wants to perform the task of chopping an onion, it has to grasp the whole onion, cut it into half, recognize its new state (sliced), grasp it accordingly, and cut it into smaller parts while continuously monitoring the state. Ultimately, the robot needs to recognize the desired state and understand when it has reached the end of the procedure (e.g. chopping). The problem of states has been analyzed in several previous works [4], [5], [6]. Similar to [6] we will address the issue of states in cooking related images.