Semantic Segmentation Using Transfer Learning on Fisheye Images

While semantic segmentation has been extensively studied in the realm of regular perspective images, its application to fisheye images remains relatively unexplored. Existing literature on fisheye semantic segmentation mostly revolves around multi-task or multi-modal models, which are computationally intensive. This motivated us to assess the performance of current segmentation methods, specifically on fisheye images. Surprisingly, we discover that these methods do not yield satisfactory results when directly trained on fisheye datasets using a fully supervised approach. This can be attributed to the fact that the models are not designed to handle fisheye images, and the available fisheye datasets are not sufficiently large to effectively train complex models. To overcome these challenges, we propose a novel training method by employing Transfer Learning (TL) on existing semantic segmentation models that concentrate on a single task and modality. To achieve this, we investigate six different fine-tuning configurations using the WoodScape fisheye image segmentation dataset. Furthermore, we introduce a pre-training stage that learns from perspective images by applying a fisheye transformation before employing transfer learning. As a result, our proposed training pipeline demonstrates a remarkable 18.29% improvement in mean Intersection over Union (mIoU) compared to directly adopting the best existing segmentation methods for fish eye images.

I. Introduction

Semantic segmentation, a popular task in computer vision, is gaining increasing popularity in the domain of fisheye images, particularly in the context of autonomous driving. Fisheye images possess a wide field-of-view (FOV), ranging from 100° to 180°, allowing them to capture a larger amount of information from the surrounding environment. This charac-teristic makes fisheye images widely utilized in autonomous driving [1], surveillance [2], and augmented reality (AR) [3] applications. However, the advantage of fisheye images comes at the cost of significant optical distortion caused by the highly non-linear mapping of real-world scenes captured by fisheye lenses [4]. Correcting this radial distortion introduces various drawbacks, such as reduced field-of-view and resampling of distorted features in the periphery [5]. Consequently, previous approaches that involved unwrapping fisheye images into rectilinear images for semantic segmentation have not yielded satisfactory performance [6], [7].