Learning an Interpretable Stylized Subspace for 3D-Aware Animatable Artforms

Throughout history, static paintings have captivated viewers within display frames, yet the possibility of making these masterpieces vividly interactive remains intriguing. This research paper introduces 3DArtmator, a novel approach that aims to represent artforms in a highly interpretable stylized space, enabling 3D-aware animatable reconstruction and editing. Our rationale is to transfer the interpretability and 3D controllability of the latent space in a 3D-aware GAN to a stylized sub-space of a customized GAN, revitalizing the original artforms. To this end, the proposed two-stage optimization framework of 3DArtmator begins with discovering an anchor in the original latent space that accurately mimics the pose and content of a given art painting. This anchor serves as a reliable indicator of the original latent space local structure, therefore sharing the same editable predefined expression vectors. In the second stage, we train a customized 3D-aware GAN specific to the input artform, while enforcing the preservation of the original latent local structure through a meticulous style-directional difference loss. This approach ensures the creation of a stylized sub-space that remains interpretable and retains 3D control. The effectiveness and versatility of 3DArtmator are validated through extensive experiments across a diverse range of art styles. With the ability to generate 3D reconstruction and editing for artforms while maintaining interpretability, 3DArtmator opens up new possibilities for artistic exploration and engagement.

I. Introduction

Artistic works serve as a reflection of artists’ inspirations and perspectives on life, expressed in a creative and imaginative manner that has evolved alongside human history. While these artworks encapsulate profound introspections, subjective interpretations of existence, and fervent admiration for life, the stories they tell remain unheard. As a result, traditional static artworks often fail to resonate with ordinary viewers, hindering their ability to appreciate the inner spirit and beauty encapsulated within. However, in the era of deep learning, advancements in technology have made it more accessible for regular users to customize and stylize artistic content. This transformative process is gradually reshaping the way people create, consume, and share art, from everyday applications such as style image manipulation to virtual concepts like the metaverse. Consequently, there is a growing demand to rebuild the interactions between artists and audiences. This leads us to an intriguing question: “Can we breathe life into existing artworks, facilitating more vivid and resonating human-art interactions, all while preserving their original content and style?” This question gives rise to the problem of “3D Animatable Artforms”, which involves the extraction of three-dimensional facial information from the given artistic face images while simultaneously maintaining the integrity of their content and style.