Design and Implementation of a Vision- and Grating-Sensor-Based Intelligent Unmanned Settlement System

In this article, a new vision- and grating-sensor-based intelligent unmanned settlement (IUS) system is proposed for convenience stores to automatically recognize the shopping behavior of customers, record their identities, and generate invoices. First, we design a new IUS architecture, which includes a shelf module and exit module. To achieve automatic settlement for each customer, a shopping event detection method is proposed. In this method, a vision-based human pose estimation algorithm is used to detect a human form standing in front of a shelf. The hand actions of each customer are detected by a grating sensor, and an image recognition method based on a convolutional neural network (CNN) is applied to recognize the items in the hands of customers. To reduce the image annotation workload, we propose a semisupervised training method for the recognition network. Based on hand action detection and item recognition, a shopping event recognition method is designed for the system, and a facial image of the customer corresponding to each shopping behavior is captured. Finally, each detected shopping event is added to the invoice of the corresponding customer via a facial recognition method. To verify the effectiveness of the proposed IUS system, we have built a handheld item image dataset and a shopping event dataset for an unmanned convenience store. The experimental results show that the proposed system can accurately recognize shopping behaviors and generate invoices.