A Deep Learning Framework for Start–End Frame Pair-Driven Motion Synthesis

A start–end frame pair and a motion pattern-based motion synthesis scheme can provide more control to the synthesis process and produce content-various motion sequences. However, the data preparation for the motion training is intractable, and concatenating feature spaces of the start–end frame pair and the motion pattern lacks theoretical rationality in previous works. In this article, we propose a deep learning framework that completes automatic data preparation and learns the nonlinear mapping from start–end frame pairs to motion patterns. The proposed model consists of three modules: action detection, motion extraction, and motion synthesis networks. The action detection network extends the deep subspace learning framework to a supervised version, i.e., uses the local self-expression (LSE) of the motion data to supervise feature learning and complement the classification error. A long short-term memory (LSTM)-based network is used to efficiently extract the motion patterns to address the speed deficiency reflected in the previous optimization-based method. A motion synthesis network consists of a group of LSTM-based blocks, where each of them is to learn the nonlinear relation between the start–end frame pairs and the motion patterns of a certain joint. The superior performances in action detection accuracy, motion pattern extraction efficiency, and motion synthesis quality show the effectiveness of each module in the proposed framework.

I. Introduction

Human motion capture data, which precisely record human motion, have been widely used as a driver for many applications, such as movie production [1], medical rehabilitation [2], [3], and humanoid robots [4], [5], but the high cost of motion capture and performance limitations of actors hinder wider applications of them. Motion synthesis technologies [6], [7], [8], [9] that can generate motion data without motion capture have drawn much research attention. However, from the conventional motion representation (global joint positions or rotations), many current motion synthesis methods [6], [7], [10] encapsulate high degrees of freedom (DOFs) of the motion data to avoid breaking the naturalness of human motion. This limits the precise control of the motion synthesis process and the diversity of generated motions.