Real-Time Monitoring System for Detecting Abnormal Operation Procedures

In this paper, we propose a task monitoring system that exemplifies man-machine collaboration in a small workshop to make real-time judgments and monitor whether the operator’s operation procedures comply with the pre-defined rules. The proposed system comprises an image target detection module, a hand action recognition module, and a procedure comparison module, where YOLOv4 is used to detect relevant objects in the scene, and SlowFast is used to recognize hand actions so as to create an action base to describe the corresponding relationship between the object and action. A Standard Operating Procedures (SOP) with pre-established rules is then used to compare against the action base established by the proposed system to check if the operation procedure flow of the human operator complies with the rules.