Action Quality Assessment Using Siamese Network-Based Deep Metric Learning

Automated vision-based score estimation models can be used to provide an alternate opinion to avoid judgment bias. Existing works have learned score estimation models by regressing the video representation to ground truth score provided by judges. However, such regression-based solutions lack interpretability in terms of giving reasons for the awarded score. One solution to make the scores more explicable is to compare the given action video with a reference video, which would capture the temporal variations vis-á-vis the reference video and map those variations to the final score. In this work, we propose a new action scoring system termed as Reference Guided Regression (RGR), which comprises (1) a Deep Metric Learning Module that learns similarity between any two action videos based on their ground truth scores given by the judges, and (2) a Score Estimation Module that uses the first module to find the resemblance of a video with a reference video to give the assessment score. The proposed scoring model is tested for Olympics Diving and Gymnastic vaults and the model outperforms the existing state-of-the-art scoring models.

I. Introduction

Humans strive to attain perfection and efficiency in their day-to-day activities and maximize the outcome of their actions. They improve their performances by comparing themselves with others or seeking feedback from experts. The task of Action Quality Assessment (AQA) involves quantifying the quality of action, i.e., determining how well an actor performs. Unlike the traditional action recognition/classification task widely attempted in the computer vision community, where a subset of discriminative frames can help identify the type of action, the task of action assessment is more challenging. It requires a quantitative and qualitative assessment of the entire action sequence. AQA has applications in many areas such as sports [15], [16], health-care [23], rehabilitation [14], exercise [6], [7] and actions of daily living [3].