Topic Embedded Representation Enhanced Variational Wasserstein Autoencoder for Text Modeling

Variational Autoencoder (VAE) is now popular in text modeling and language generation tasks, which need to pay attention to the diversity of generation results. The existing models are insufficient in capturing the built-in relationships between topic representation and sequential words. At the same time, there is a massive contradiction between the commonly used simple Gaussian prior and the actual complex distribution of language texts. To address the above problems, we introduce a hybrid Wasserstein Autoencoder (WAE) with Topic Embedded Representation (TER) for text modeling. TER is obtained through an embedding-based topic model and can capture the dependencies and semantic similarities between topics and words. In this case, the learned latent variable has rich semantic knowledge with the help of TER and is easier to explain and control. Our experiments show that our method is competitive with other VAEs in text modeling.