Sky Optimization: Semantically aware image processing of skies in low-light photography

The sky is a major component of the appearance of a photograph, and its color and tone can strongly influence the mood of a picture. In nighttime photography, the sky can also suffer from noise and color artifacts. For this reason, there is a strong desire to process the sky in isolation from the rest of the scene to achieve an optimal look. In this work, we propose an automated method, which can run as a part of a camera pipeline, for creating accurate sky alpha-masks and using them to improve the appearance of the sky. Our method performs end-to-end sky optimization in less than half a second per image on a mobile device. We introduce a method for creating an accurate sky-mask dataset that is based on partially annotated images that are inpainted and refined by our modified weighted guided filter. We use this dataset to train a neural network for semantic sky segmentation. Due to the compute and power constraints of mobile devices, sky segmentation is performed at a low image resolution. Our modified weighted guided filter is used for edge-aware upsampling to resize the alpha-mask to a higher resolution. With this detailed mask we automatically apply post-processing steps to the sky in isolation, such as automatic spatially varying white-balance, brightness adjustments, contrast enhancement, and noise reduction.

1. Introduction

Professional photographers generally invest time post-processing the appearance of the sky, as it significantly affects how humans perceive the photograph’s time of day, and the relative appearance of the non-sky foreground of the scene. Photographers will often manually segment the sky and use that segmentation to adjust the sky’s brightness, contrast, color, and noise properties. This editing is particularly necessary in night-time scenes, wherein the camera receives little light and therefore produces images with significant noise. Noise in the sky can look particularly unattractive and noticeable because the sky is typically textureless. Additionally, night-time scenes may contain a foreground that is illuminated by a nearby light source, while the sky is illuminated by scattered sunlight or by distant terrestrial lights reflected off of clouds. This means that the standard practice of using a single illuminant estimate for white balance [1] is physically incorrect, and results in an unnatural tint of either the sky or of the foreground. This motivates our use of sky segmentation for performing spatially-varying white balance, which ameliorates this issue.