AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration

Diffusion models are emerging expressive generative models, in which a large number of time steps (inference steps) are required for a single image generation. To accelerate such tedious process, reducing steps uniformly is considered as an undisputed principle of diffusion models. We consider that such a uniform assumption is not the optimal solution in practice; i.e., we can find different optimal time steps for different models. Therefore, we propose to search the optimal time steps sequence and compressed model architecture in a unified framework to achieve effective image generation for diffusion models without any further training. Specifically, we first design a unified search space that consists of all possible time steps and various architectures. Then, a two stage evolutionary algorithm is introduced to find the optimal solution in the designed search space. To further accelerate the search process, we employ FID score between generated and real samples to estimate the performance of the sampled examples. As a result, the proposed method is (i).training-free, obtaining the optimal time steps and model architecture without any training process; (ii). orthogonal to most advanced diffusion samplers and can be integrated to gain better sample quality. (iii). generalized, where the searched time steps and architectures can be directly applied on different diffusion models with the same guidance scale. Experimental results show that our method achieves excellent performance by using only a few time steps, e.g. 17.86 FID score on ImageNet 64 × 64 with only four steps, compared to 138.66 with DDIM.

1. Introduction

Diffusion models are a class of generative models that exhibit remarkable performance across a broad range of tasks, including but not limited to image generation [14], [24], [8], [2], [29], [4], [15], [38], super-resolution [33], [39], [6], inpainting [22], [31], and text-to-image generation [25], [32], [27], [10]. These models utilize the diffusion process to gradually introduce noise into the input data until it conforms to a Gaussian distribution. They then learn the reversal of this process to restore the data from sampled noise. Consequently, they achieve exact likelihood computation and excellent sample quality. However, one major drawback of diffusion models is their slow generation process. For instance, on a V100 GPU, generating a 256 × 256 image with StyleGAN [16] only takes 0.015s, whereas the ADM model requires multiple time steps for denoising during generation, leading to a significantly longer generation time of 14.75s.