Using Gaussian Process in Clockwork Variational Autoencoder for Video Prediction

Accurately predicting long video sequences has been an open challenge for the machine learning research community, and we try to address it in this study. We present Gaussian Process Clockwork Variational Autoencoder (GPC-VAE), a generative model which combines the Gaussian Process with the Clockwork Variational Autoencoder (CW-VAE). We use this new variant of the VAE model for unsupervised learning on the video prediction task. We conduct a series of experiments on MineRL and MMNIST, two publicly available datasets consisting of video sequences. We evaluate our model using two well-known image quality metrics and discuss its performance compared to the benchmarks in video prediction. The results indicate that our model can outperform the baselines of video prediction in certain scenarios, but the generalizability of our generative model still needs to be improved.