SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors

We propose SceneTex, a novel method for effectively gen-erating high-quality and style-consistent textures for indoor scenes using depth-to-image diffusion priors. Unlike pre-vious methods that either iteratively warp 2D views onto a mesh surface or distillate diffusion latent features with-out accurate geometric and style cues, SceneTexformulates the texture synthesis task as an optimization problem in the RGB space where style and geometry consistency are prop-erly reflected. At its core, SceneTex proposes a multires-olution texture field to implicitly encode the mesh appear-ance. We optimize the target texture via a score-distillation-based objective function in respective RGB renderings. To further secure the style consistency across views, we introduce a cross-attention decoder to predict the RGB values by cross-attending to the pre-sampled reference locations in each instance. SceneTex enables various and accurate texture synthesis for 3D-FRONT scenes, demonstrating sig-nificant improvements in visual quality and prompt fidelity over the prior texture generation methods.

1. Introduction

Synthesizing high-quality 3D contents is an essential yet highly demanding task for numerous applications, including gaming, film making, robotic simulation, autonomous driving, and upcoming VR/AR scenarios. With an increasing number of 3D content datasets, the computer vision and graphics community has witnessed a soaring research inter-est in the field of 3D geometry generation [2], [12], [36], [38], [40], [60], [68], [73]. Despite achieving a remarkable success in 3D geometry modeling, generating the object appearance, i.e. textures, is still bottlenecked by laborious human efforts. It typically requires a substantially long time for designing and adjustment, and immense 3D modelling expertise with tools such as Blender. As such, automatic designing and augmenting the textures has not yet been fully industrial-ized due to a huge demand for human expertise and finan-cial expenses. Figure 1.

We introduce SceneTex, a text-driven texture synthesis architecture for 3D indoor scenes. Given scene geometries and text prompts as input, SceneTex generates high-quality and style-consistent textures via depth-to-image diffusion priors.