Correcting aspect ratio distortion of natural images by convolutional neural network

We propose a method to correct the aspect ratio distortion of images using convolutional neural network (CNN). The “aspect ratio”, which is focused on this research, is defined as the relative “image aspect ratio” (i.e. ratio of width to height of image) from non-stretched image. And the aspect ratio can be distorted by vertical or horizontal stretching, which does not maintain the image aspect ratio. In the proposed method, we construct an aspect ratio estimator whose input is a (possibly distorted) image and output is a scalar value of aspect ratio. Since estimation of aspect ratio from image can be regarded as regression problem, we modeled the estimator by CNN. Once we have a reliable estimate of aspect ratio of an image, the correction can be done straightforwardly by inverse stretching. In the experiments, we evaluated performance of the model trained on Pascal VOC2012 natural image dataset. Our method can accurately correct the distortion within 7% of stretch from original images on average, which outperforms average human performance (i.e. about 13%). In terms of precision, 85% of distorted images are successfully corrected. We also experimented to confirm what information is used by trained CNN. As the result, we observed that the model uses the edge component of the image to estimate the aspect ratio.