BokehMe: When Neural Rendering Meets Classical Rendering

We propose BokehMe, a hybrid bokeh rendering framework that marries a neural renderer with a classical physically motivated renderer. Given a single image and a potentially imperfect disparity map, BokehMe generates high-resolution photo-realistic bokeh effects with adjustable blur size, focal plane, and aperture shape. To this end, we analyze the errors from the classical scattering-based method and derive a formulation to calculate an error map. Based on this formulation, we implement the classical renderer by a scattering-based method and propose a two-stage neural renderer to fix the erroneous areas from the classical renderer. The neural renderer employs a dynamic multi-scale scheme to efficiently handle arbitrary blur sizes, and it is trained to handle imperfect disparity input. Experiments show that our method compares favorably against previous methods on both synthetic image data and real image data with predicted disparity. A user study is further conducted to validate the advantage of our method.

1. Introduction

Bokeh effect refers to the way the lens renders the out-of-focus blur in a photograph (Fig. 1). With different lens designs and configurations, various bokeh styles can be created. For example, the shape of the bokeh ball can be controlled by the aperture. Classical rendering methods [6], [20], [31], [40] can change bokeh styles easily by controlling the shape and size of the blur kernel. However, they often suffer from artifacts at depth discontinuities. Neural rendering methods [11], [25], [32] can address this problem well by learning from image statistics, but they have difficulty simulating real bokeh balls and can only produce the bokeh style from the training data. In addition, previous neural rendering methods lack a mechanism to produce large blur size on high-resolution images, because of the fixed receptive field of the neural network and the blur size limit of the training data.