Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360° Image Outpainting

360° images, with a field-of-view (FoV) of $180^{\circ}\times 360^{\circ}$, provide immersive and realistic environments for emerging virtual reality (VR) applications, such as virtual tourism, where users desire to create diverse panoramic scenes from a narrow FoV photo they take from a viewpoint via portable devices. It thus brings us to a technical challenge: ‘How to allow the users to freely create diverse and immersive virtual scenes from a narrow FoV image with a specified viewport?’ To this end, we propose a transformer-based 360° image outpainting framework called Dream360, which can generate diverse, high-fidelity, and high-resolution panoramas from user-selected viewports, considering the spherical properties of 360° images. Compared with existing methods, e.g., [3], which primarily focus on inputs with rectangular masks and central locations while overlooking the spherical property of 360° images, our Dream360 offers higher outpainting flexibility and fidelity based on the spherical representation. Dream360 comprises two key learning stages: (I) codebook-based panorama outpainting via Spherical-VQGAN (S-VQGAN), and (II) frequency-aware refinement with a novel frequency-aware consistency loss. Specifically, S-VQGAN learns a sphere-specific codebook from spherical harmonic (SH) values, providing a better representation of spherical data distribution for scene modeling. The frequency-aware refinement matches the resolution and further improves the semantic consistency and visual fidelity of the generated results. Our Dream360 achieves significantly lower Frechet Inception Distance (FID) scores and better visual fidelity than existing methods. We also conducted a user study involving 15 participants to interactively evaluate the quality of the generated results in VR, demonstrating the flexibility and superiority of our Dream360 framework.

1 Introduction

The ability to observe complete surroundings in one shot and choose any viewport for immersive experience (see Fig. 1(a)) has led to the growing popularity of 360° images (a.k.a. panoramas)¹

With equirectangular projection (ERP) images by default.

flourishing virtual reality (VR) applications [4], [5], [30], [35], [36], [52], [58], such as virtual tourism. In this case, users wish to create diverse virtual panoramic scenes from a narrow FoV (NFoV) photo they take from a viewpoint via portable devices (e.g., smartphones). This way, it offers the users personalized contents and enables them to perceive and interact with virtual environments. This motivates us to study diverse high-fidelity and high-resolution panorama generation while users provide NFoV images at desired viewpoints.