Application of Semantic Image Segmentation for Efficient UAS-SfM Photogrammetry Mapping

Structure from Motion (SfM) photogrammetry in combination with the Multi-View Stereo (MVS) technique (SfM-MVS ) is a low-cost and effective tool to reconstruct the 3D structure of the real-world environments and objects using a set of overlapping images. This is approach is often implemented using imagery collected by a small uncrewed aircraft system (UAS) equipped with a high-resolution digital camera (referred to as UAS-SfM). The generated dense point cloud is typically considered the ultimate product of the workflow in almost all commercial and open-source software packages. Although, currently, UAS photogrammetry is being widely used in many remote sensing (RS) applications, there are some challenges regarding the efficiency, quality, and computational complexity in deriving some geospatial products in certain environments, especially where frequent mapping of the area is required to monitor certain changes in the surveyed environment. Efficient mapping of the coastal environments can be challenging due to the presence of water or other moving objects in almost all UAS images. It may also be challenging due to the need for accurate classified 2D and 3D land cover maps or digital terrain models (DTMs) which are required to examine the rate of changes in those environments. Providing those products through the SfM-MVS photogrammetry requires processing a large number of UAS images as well as performing expensive and complicated processing on 2D images and/or 3D point clouds which leads to an exponential increase in the computational complexity and cost in coastal mapping. This study proposes a novel approach that integrates semantic UAS image segmentation into the SfM-MVS photogrammetry workflow to address the aforementioned challenges in UAS photogrammetry mapping in coastal environments. The proposed approach leads to a higher level of automation in generating geospatial products with efficient exploitation of the available computation resources for SfM-MVS computations. The proposed approach exploits state-of-the-art deep learning (DL) for semantic UAS image segmentation to predict land cover image masks. Image masks predicted for moving objects, present in the study area, e.g., water bodies, are used to automatically exclude those areas from SfM-MVS computation. That leads to a higher computation efficiency and an lower noise in the reconstructed model. The proposed approach is then expanded to predict image mask through the DL-based image segmentation model for an automatic generation of the classified point cloud simultaneous with the SfM-MVS processing for reconstruction.

I. Introduction

In the last few years, the uncrewed aircraft systems (UASs) equipped with high-resolution cameras have been used as one of the most inexpensive, yet effective tools in remote sensing (RS) for survey-grade mappings in local scales [5]. Structure-from-Motion (SfM) Photogrammetry technique, as a highly-efficient alternative to the traditional digital photogrammetry, is employed to derive the 3D structure of the surveyed area with fine details using a sequence of overlapping images collected by a high-resolution camera onboard the UAS flying at a low altitude. The geometry of the 3D scene, including the accurate position and orientation of the camera at each exposure station, the internal geometry of the camera, and a sparse point cloud representing the 3D structure of the surveyed area and objects are the main products of the SfM computations in almost all open-source and commercial SfM software packages. SfM photogrammetry workflow is usually paired with multi-view stereo (MVS) algorithms which leads to the generation of a dense point cloud representing the study area reconstructed with fine detail. SfM-MVS photogrammetry is usually referred to as a roughly automated approach to derive 2D and 3D geospatial information from a set of overlapping uncalibrated images acquired from the study area.