Skeleton recognition-based motion generation and user emotion evaluation with in-home rehabilitation assistive humanoid robot

The shortage of nurses and the increasing elderly population demand robots in nursing that can carry out care tasks safely and intelligently. In this study, the method of skeleton recognition-based motion generation of the humanoid robot for the human range-of-motion training with dual 7-DOF arm manipulation is developed. Mediapipe-based skeleton recognition is installed with humanoid robot to recognize the human pose even though the whole of body is not seen by a camera. The 7-DOF arm was controlled to reach the detected 3D coordinates of the human right shoulders. In the experiment, the robot stood at three positions: where experimental partici-pants could see the robot fully, where the participants could see the robot partially, and where the participants could not see the robot. In each standing point, the robot uses one arm to reach to the human's shoulder with 3 patterns of waypoints while the other hand supports the human's hand. The system successfully generated the motion for the mentioned conditions except when human had his/her back towards the robot. Results show that it was difficult to recognize human body parts when the back view of the participants could only be partially captured. In terms of motion generation, robot needs to stand in front or sideway of people for reaching hand to conduct range-of-motion training. In addition, we assume that upper waypoint has a relatively high acceptance when the participants did not look at the robot fully (the condition where robot stands in sideway of the human).