A Novel Marine Oil Spillage Identification Scheme Based on Convolution Neural Network Feature Extraction From Fully Polarimetric SAR Imagery

Marine oil spill pollution has caused serious impacts on marine ecological environments, ecological resources and marine economy. Synthetic Aperture Radar (SAR), especially polarmetric SAR (PolSAR), has been proven to be a powerful and efficient tool for marine oil spill detection. In general, traditional oil spill detection methods mainly rely on artificially-extracted polarization characteristics, and the detection accuracy is limited by the quality of feature extraction. Recently proposed Convolutional neural network (CNN) is capable of mining spatial feature from large data set automatically. Inspired by these, in this paper we proposed a novel oil spill identification method based on multi-layer deep feature extraction by CNN. Firstly, PolSAR data are converted into a 9-channel data block to feed the CNN. Then, a 5-layer CNN architecture is built to extract two high-level features from the original data automatically. The features are fused after dimension reduction via principal component analysis (PCA). Finally, support vector machine method with radial basis function kernel (RBF-SVM) is utilized for classification. Three sets of RADARSAT-2 fully polarimetric SAR data were used in this study to validate the proposed method. The obtained results reveal that the proposed method provides competitive results in overall classification accuracy and kappa coefficient. Moreover, this method can improve the accuracy of oil spill detection, reduce the false alarm rate, and effectively distinguish an oil spill from a biogenic slick.

SECTION I. Introduction

Marine oil spills often result in large-scale marine pollution and seriously endangers the marine ecosystems and environment, fisheries, wildlife and other social interests. It is of great significance to perform timely and effective marine oil spill monitoring [1]–​[3]. Among the available methods, Synthetic Aperture Radar (SAR) has become the main means of oil spill detection, as it is minimally affected by sunshine and clouds, and has all-day and all-weather imaging capability with wide swath and high resolution [4], [5].

In recent years, with the development of polarimetric SAR technology, polarimetric SAR (PolSAR) images are acquired by transmitting and receiving electromagnetic (EM) waves in different polarizations [6], [7]. PolSAR imagery provides not only the scattering mechanism of sea surface but also a complex coherency matrix [8], scattering matrix [8], and other polarimetric information for oil spill detection [9]. PolSAR oil spill detection of the sea surface has become a research hotspot in recent years. Many studies have confirmed the effectiveness of PolSAR in detecting oil spills at the sea surface [10], [11]. However, the automatic extraction and selection of features of objects in PolSAR image target detection and classification have proven to be a long-term challenge [4], [12]. Based on the EM scattering characteristics and mechanism of targets, different PolSAR features of oil spill detection methods were proposed in the literature. Migliaccio et al. [13] introduced polarimetric entropy (H), mean scattering angle (a) and anisotropy (A) into oil spill detection based on target decomposition theory. Their results showed that, under low wind speed, the polarimetric features of H and A can effectively distinguish oil spills from sea water, and the polarimetric entropy has a better distinguishing effect. Later, the validity of co-polarized phase difference (CPD) in oil spill detection was proven [14]. Zhang et al. [15] calculated the conformity coefficient ($\mu $ ) from fully polarimetric SAR data to map an oil spill in the Gulf of Mexico under the conditions of low and medium wind speeds, and the results showed strong oil spill detection results. Using SIR-C multi-polarization data, Duan and Chong [16] proposed a method to distinguish mineral oil from biogenic oil in multi-polarization SAR images based on the cross-polarization ratio. Wenguang et al. [6] combined four common polarimetric features into a new feature to extract oil spill information. Skrunes et al. [17] comprehensively analyzed various polarimetric features, such as geometric intensity, and the real part of the co-polarization cross product ($r_{CO}$ ), and detected oil spills through feature combination. Song et al. [18] summarized and analyzed various polarimetric features and pointed out that different features exhibit different responsiveness to different small targets at sea. There are many other features, such as degree of polarization (DoP) [19], Span [20], Muller filtering method [21], and Pedestal height [22] that were used effectively in oil spill detection. In addition, texture features, context features and statistical features are also used to enhance PolSAR image classification [23]. On the other hand, machine learning methods such as support vector machine (SVM) [24], decision tree [25], fuzzy logic algorithm [26] and neural networks [27]–​[29] have also been used in PolSAR oil spill detection. Frate et al. [27] first applied the multi-layer perceptron (MLP) neural network method to SAR images to recognize and extract oil spills, and achieved good results, which laid a foundation for the application of the neural network method in SAR oil spill detection. In view of the commonly used oil spill features, Xu et al. [30] performed a detailed analysis and evaluation of seven classical oil spill detection methods, and studied the application situation of different oil spill detection methods and the required combination of features.

In general, the accuracy of classification is greatly dependent on the quality of extracted features. Artificial feature extraction is complex and time-consuming, which usually takes significant efforts and requires deep domain knowledge [31], [32]. In addition, the extracted features may have incomplete information coverage or redundancy [23], thus resulting in poor classification performance. In addition, different classifiers also have certain performance differences for different features or feature combinations [23], [30]. Traditional feature extraction and pixels-based classification methods are often affected by severe speckle noise, which leads to a high false alarm rate [33], [34]. These problems increase the difficulty of oil spill detection at sea [35]. It would be important to comment that another way to classify this type of data is using region-based algorithms, which classify the images using regions, not pixels, as processing unities [48], [68], [71]. These studies all prove that the spatial information can suppress speckle noises and improve the accuracy of classification.

The application of deep learning in optical image [36], [37] and SAR imagery for target recognition [34] has achieved some degree of success. The recognition technology based on deep learning has the abilities of multi-level feature expression and non-linear data fitting [38]. It can automatically mine more discriminant and representative features, and thus enhance classification performance. The Convolutional Neural Network (CNN), which is one of the widely used models in deep learning [39], can mine the spatial correlation of data to reduce the number of trainable parameters in the network. Chen et al. [40] studied the SAR target recognition by using the CNN, and proposed a full convolutional network structure (A-ConcNets) to effectively prevent over-fitting issue that comes with every Neural Network training. Zhou et al. [34] converted the complex covariance matrix of PolSAR image into a six-channel real matrix to adapt to the input of the network, and then input the six-channel data into the CNN for PolSAR classification. Gao et al. [35] proposed a novel method based on a dual-branch deep CNN (Dual-CNN) to improve the classification of PolSAR images.

Although the deep learning framework can effectively mine the rich features of PolSAR data, its research in PolSAR oil spill detection is still at its early stage, and there are issues on how to effectively integrate multi-layer deep features [41] and improve the performance of weak classifiers in CNN [42].

In view of the current research situation and exiting problems of PolSAR oil spill detection mentioned above, in this paper a novel oil spill detection method based on deep learning feature fusion and SVM is proposed. The main work includes the following three aspects:

1. a deep CNN network suitable for oil spill detection using PolSAR data is constructed, which consists of three convolution layers and two pooling layers. It is used to automatically extract representative deep features from PolSAR data without manual feature extraction and selection before classification.

2. On the basis of the traditional CNN structure, this study further extracted two high-level features from the network. Then, by means of the principal component analysis (PCA) dimensionality reduction method, the two features were fused together in order to make full use of the depth features extracted by the network. In addition, during the final stages, the features were visualized following the completion of the dimensionality reduction process, and the representativeness of the features was found to be more intuitively displayed.

3. Finally, the fused features are input into the SVM classifier to improve the final classification performance. The experimental results show that the algorithm performs well in terms of classification accuracy and Kappa coefficient, and can achieve good oil spill detection results.

The remainder of this paper is organized as follows: In Section 2, we briefly introduce the RADARSAT-2 PolSAR data used in this study and the proposed approach in detail. The experimental results and comparative analysis are given in Section 3. In Section 4, the depth features are visualized and we provide justification on superiority of the proposed method over the traditional methods. Finally, the conclusion is in Section 5.

SECTION II. Data and Methodology

A. Remote Sensing Data

Figure 1 shows the three sets of RADARSAT-2 fine-mode full-polarization polarimetric SAR images used in this study. They are SLC (single-look complex) images each with four polarimetric channels (HH, VV, HV, VH). RADARSAT-2 has a very low noise floor of about −35 dB [15]. The first two images were acquired in the Gulf of Mexico. Figure 1(a) is the first set of data, the image coverage of which is $32.95\times23.2$ km. The space resolutions along the azimuth and range directions are 4.72 m and 4.78 m, respectively, and the incident angle of the image is 26.093° ~ 29.395°. Figure 1(b) shows another image of the RADARSAT-2 spill captured in the Gulf of Mexico on June 17, 2011, of which the coverage area is 37.17 $\times$ 19.34 km, the azimuth resolution is 4.73 m, the range resolution is 5.05 m, and the range of incidence angle in this image is 43.631° ~ 44.954°. Figure 1(c) is the image that was acquired by the Norwegian Clean Seas Association for Operating Companies (NOFO), in the North Sea of Europe from the oil spill experiments on June 8, 2011. The longitude and latitude coordinates of the image center point are 2.35°E and 60.02°N. In this oil spill experiment, three different substances were released, i.e. crude oil, emulsified oil and plant oil [17].

FIGURE 1.

Marine oil spill images of the RADARSAT-2 used in this study: (a) Dataset 1; (b) Dataset 2; (c) Dataset 3.

The red box in the Figure 1 is the experimental area, the size of the subsets extracted from the images in terms of pixels and more detailed data imaging parameters are shown in Tables 1–​3. As can be seen from the PolSAR oil spill image of Figure 1, the oil spill shows as dark spots on the SAR image. There are also some other oceanic phenomena or targets present in the images, such as ships, ship wakes, drilling platforms, and so on.

TABLE 1 Imaging Information From the Radarsat-2 Used in This Study (Dataset 1)

TABLE 2 Imaging Information From the Radarsat-2 Used in This Study (Dataset 2)

TABLE 3 Imaging Information From the Radarsat-2 Used in This Study (Dataset 3)

B. Basic Framework of the CNN

In most cases, the CNN is composed by the stacking of the input layer, convolution layer, pooling layer and fully connected layer [43]. All layers are then connected in series, and the input data of each layer are the output data of the former layer. The input layer is used to receive image data. Due to its hierarchical connection structure, the CNN can extract high-level features from low-level features [44].

The convolution layer is also called the feature extraction layer. The feature map [45] is obtained by convoluting the input data. If the $l^{th}$ layer is the convolution layer, then the formula for calculating the $j^{th}$ feature map of the $l^{th}$ layer is as follows:\begin{equation*} x_{j}^{l} =f\left ({{\sum \limits _{i\in M_{j}} {x_{j}^{(l-1)}} \ast k_{ij}^{l} +b_{j}^{l}} }\right)\tag{1}\end{equation*} where $x_{i}^{(l-1)} $ indicates the $i^{th}$ feature map of the $l-1$ layer; $M_{j} $ indicates the feature map of input; * is the convolution operation; $x_{j}^{l} $ indicates the output feature map obtained after convolution operation; $k_{ij}^{l} $ is the convolution kernel between the $i^{th}$ feature map of the $l-1$ layer and the $j^{th}$ feature map of the $l$ layer; $b_{j}^{l} $ is the bias; and $f(\cdot)$ is the nonlinear activation function, such as the sigmoid function and rectification linear unit (ReLU) [46].

The convolution layer is typically followed by the pooling layer. The pooling layer can reduce the size of feature map and prevent over-fitting. Common pooling operations include max pooling, average pooling, and so on [47]. In this paper, we used the maximum pooling method, in which the maximum value of local windows in the feature map is selected as the output. Through the alternate combination of the convolution layer and pooling layer, more advanced features can be extracted from the original data.

C. Proposed Oil Detection Algorithm

Figure 2 shows our PolSAR oil spill classification framework. PolSAR data are preprocessed to adapt to the input of the CNN network. Then the CNN network is constructed and training samples are extracted from the data to train the CNN. Next, the trained model is used to extract the deep features of PolSAR data, and PCA is introduced to reduce the dimension of features. Finally, different high-level features are fused, and the SVM classifier with Radial Basis Function (RBF-SVM) kernel is used for classification. The details of the proposed method are described in detail in the following sections.

FIGURE 2.

Flow. chart of the ocean oil spill identification in this scheme.

1) Organization Form of Polsar Data in CNN Input

Each pixel in the PolSAR data can be presented as a 2*2 scattering matrix, which contains information for describing coherence or pure scatterers [10]. Under the hypothesis of reciprocity theory, Monostatic PolSAR data can be fully represented by a symmetrical 3*3 complex coherency matrix ${\boldsymbol T}_{3} $ , which is more suitable for describing distributed scatterers and can extract a series of effective polarimetric features [48], [68]. The form of ${\boldsymbol T}_{3} $ matrix is shown in Equation (2):\begin{equation*} {\boldsymbol T}_{3} =\left [{ {{\begin{array}{ccc} {T_{11}} & {T_{12}} & {T_{13}} \\ {T_{21}} & {T_{22}} & {T_{23}} \\ {T_{31}} & {T_{32}} & {T_{33}} \\ \end{array}}} }\right]\tag{2}\end{equation*} where $T_{11},T_{22},T_{33} $ are real numbers, and the other matrix elements are complex numbers. In order to make full use of polarimetric information, we used the matrix $T_{3} $ as the input of CNN. For each pixel, the polarimetric data can be defined as a vector $t$ , which is expressed as Equation (3):\begin{align*}t=&[T_{11},T_{22},T_{33},Re(T_{12}), \\& \qquad \text{Im}(T_{12})\text{Re}(T_{13}),\text{Im}(T_{13}),\text{Re}(T_{23}),\text{Im}(T_{23})] \tag{3}\end{align*} where Re and Im represents the real and imaginary part, respectively. Therefore, the PolSAR data are constructed into a nine-channel image data, which can then be used to generate a patch as input of CNN, as shown in Figure 3.

FIGURE 3.

The nine-channel form of PolSAR data.

2) Network Structure

In PolSAR image classification, the most commonly used CNN is Lenet-5 or its improved models [32]. Other very deep CNNs, such as AlexNet [36], VGG-Net [49], and Res-Net [50], are suitable for large-scale input images. For each network, a large number of samples are required to train the parameters in the network. We divided PolSAR images into a large number of input image patches as samples, and randomly selected training samples for CNN construction and validation. The input patch size is generally small, such as $7\times 7$ , $9\times 9$ , $15\times 15$ [32]. When small input patches pass through multiple convolution and pooling layers, the size of the final feature map will be smaller. Therefore, in this paper, based on the classical Lenet-5, a CNN network suitable for feature extraction of PolSAR oil spill images is constructed as shown in Figure 4. Our CNN framework consists of three convolution, two max pooling, a reshape and a fully connected layer. A Softmax classifier is connected to the output [40].

FIGURE 4.

Structure diagram of the CNN.

The first two convolution layers consist of 30 and 60 convolution kernels with a size of $3\times 3$ , which are used to automatically extract features from the input data. The first two convolution layers are followed by a max pooling layer with a pool size of $2\times 2$ , and a step size of 2 pixels. The third convolution layer uses $1\times 1$ convolution, and the number of convolution kernels is set to 20, which is used to delete redundant features, improve the representativeness of features, and reduce the computational complexity. We apply L2 regularization [51] to each convolution layer. Then the resulting feature map is stretched into a 1-D vector for input into the fully connected network. The number of neurons in the fully connected layer is 240. The number of neurons in the final layer is the number to be classified, and we set it to 2 (or 3 in Dataset 3). Finally, the softmax classifier is used to calculate the probability of each class. The ReLU activation function is applied to all convolution layers and the fully connected layer. In order to avoid over-fitting of the network, the dropout operation [52] is added after the fully connected layer, and the dropout rate is set to 0.5. The further parameters of the designed CNN are listed in Table 4.

TABLE 4 Detailed Configuration of the Network Structure

After the forward propagation mentioned above has been completed, the loss function is then calculated based on the predicted and true value. In this study, we used cross-entropy as the loss function. The formula is as follows:\begin{equation*} L=-\frac {1}{n}\sum \limits _{i=1}^{N} {\sum \limits _{k=1}^{n} {[y\ln a+(1-y)\ln (1-a)] }}\tag{4}\end{equation*} where $N$ represents the number of training sets; $n$ indicates the number of neurons in the output layer, representing the number of categories to be classified; $y$ is the target value; and $a$ denotes the actual output value of the network.

Next, the network parameters are updated iteratively according to back propagation (BP) algorithm. In BP, the stochastic gradient descent (SGD) algorithm is used to update the network parameters along the opposite direction of the gradient of the objective function to make the objective function converge. However, the stochastic gradient descent algorithm is prone to oscillation near the local extremum, resulting in a slow convergence rate. In order to improve the performance of BP, we used the Adadelta optimization algorithm [53] in the process of gradient descent. This method adjusts the learning rate adaptively and obtains a predicted result quickly. The batch size is set to 64.

3) Extraction and Fusion of Deep Features

For the entire network, each convolution and pooling layer can be regarded as a feature extraction layer. For the new input data, the trained CNN model can automatically extract high-level features from the data. In this paper, the feature map after the third convolution layer is extracted and stretched to a 1-D vector, recorded as $fea\_{}1$ . At the same time, the features of the fully connected layer in the network are extracted and recorded as $fea\_{}2$ . These two features, including spatial and other information, are regarded as two different high-level features of input image patches. However, redundancy may be presented for these two layers of features, thus PCA [54] is introduced to reduce the dimension of the extracted features. In this paper, the first three principal components of PCA processing are taken as new features. Finally, the features of the two layers are connected in series as deep fusion features.

4) Classification of Deep Features Using RBF-SVM

Based on the network constructed in this paper, the CNN is used to extract deep features from the PolSAR images. Then we use fully connected layer and softmax layer to classify the features. This method can effectively update the parameters in the training phase by combining BP. However, Softmax is not very effective in non-linear classification [55]. Therefore, we introduce RBF-SVM into the classification scheme to improve the classification performance of the top-layer classifier of the CNN network structure [56], and at the same time, avoid the problem of over-fitting to a certain extent [57].

SVM aims at minimizing structural risk. It maps sample vectors to high-dimensional or even infinite-dimensional feature space (Hilbert space) through non-linear mapping, then constructs the optimal hyperplane as the decision plane in high-dimensional feature space [24]. In SVM, the non-linear separable problem in the original sample space is transformed into a linear separable problem in the feature space. Therefore, SVM performs well in solving the non-linear classification problem.

Assuming $T=\left \{{{(x_{1},y_{1}),(x_{2},y_{2})\ldots,(x_{n},y_{n})} }\right \}$ is the sample to be classified, the essence of the SVM algorithm is to solve the optimization problem of objective function:\begin{align*}&\textrm {min}\frac {1}{2}\left \|{ w }\right \|^{2}+C\sum \limits _{i=1}^{n} {\xi _{i}} \\&s.t. \begin{cases} {y_{i} (w\cdot \varphi (x_{i})+b)\ge 1-\xi _{i}} \\ {\xi _{i} \ge 0} \\ \end{cases};\quad i=1,2,\ldots,n\tag{5}\end{align*} where $w$ indicates the normal vector and decides the direction of hyperplane; $b$ indicates the offset and determines the distance between the hyperplane and original point; $y_{i} $ is the correct category label; $C$ represents the penalty coefficient; and $\xi _{i} $ is the coefficient of relaxation. When solving the optimization problems, the mathematical model is typically transformed into a Lagrangian dual problem.\begin{align*}&\mathop {\textrm {max}}\limits _{\alpha } \sum \limits _{i=1}^{n} {\alpha _{i}} -\frac {1}{2}\sum \limits _{j=1}^{n} {\sum \limits _{i=1}^{n} {\alpha _{i} \alpha _{j} y_{i} y_{j} k(x_{i},x_{j})}} \\&s.t. \begin{cases} {\sum \limits _{i=1}^{n} {\alpha _{i} y_{i} =0}} \\ {0\le \alpha _{i} \le C} \\ \end{cases};\quad i=1,2,\ldots,n\tag{6}\end{align*} where $\alpha _{i} $ represents the Lagrange weight, and $k(x_{i},x_{j})$ represents the kernel function of SVM. The commonly used kernel functions are linear kernel functions or RBF kernel functions [58]. RBF kernel functions are mainly used in the case of linear inseparability. Although the process of determining the appropriate parameters in RBF-SVM is time-consuming, once the appropriate parameters have been determined, the RBF-SVM classifier usually achieves better performance than linear kernel SVM. Therefore, we used RBF-SVM for classification.

SECTION III. Expermental Results and Analysis

A. Experimental Settings

In order to verify the effectiveness of the algorithm, several comparison experiments were performed. These experiments adopted three classic methods: Maximum Likelihood (ML); SVM; and Neural Networks (NN). The experiments were simultaneously performed on two types of input data. In addition, several of the most commonly used polarization features (PF) utilized in oil spill detection processes were selected as input, which included span [13], H, $\alpha $ , $\mu $ [9], $Dop$ [12], and the feature images corresponding to Dataset 1, as detailed in Fig. 5. The classical algorithm ML was performed on the ENVI remote sensing image processing platform [64]. The neural network was implemented on a MATLAB platform, including an input layer, single hidden layer, and output layer. The number of neurons in the input layer was 9, which corresponded to vector $t$ in Eq. (3). The number of hidden layer neurons was set as 64, and output layer neurons corresponded to the number of categories. The initial learning rate was set at 0.01 and the maximum number of iterations was 100. In the current study, since the LIBSVM library (http://www.csie.ntu.edu.tw/~cjlin/libsvm/) is considered to be a mature SVM framework [67], the LIBSVM was adopted on Matlab R2018b for the purpose of implementing the support vector machine portion of the experiment. The parameters of the support vector machine classifier included a penalty coefficient $C$ and a Gaussian kernel parameter $\sigma $ . In addition, a 5-fold cross-validation method in LIBSVM was employed to determine the optimal $C$ and $\sigma $ , and all of the optimal $C$ and $\sigma $ were used in following experiments. Also, two traditional CNN-based methods were compared using the T3 matrix and PF as the inputs, respectively, and were denoted in this study as T3-CNN and PF-CNN. Then, based on this study’s proposed experimental scheme, the obtained results were denoted as PF-CNN+SVM and T3-CNN+SVM, respectively. The specific experimental results and analyses processes are described in detail in a following section of this study.

FIGURE 5.

Polarimetric feature images of the Dataset 1.

In the present experimental study, in accordance with the network structure design described in Section 2.3.2, the areas surrounding the pixels were selected as the input image patches. The image patched measured $15\times 15\times 9$ and included nine channels. Then, a label was assigned for each image patch. Specifically, the location of the center pixel of the image patch was chosen as the index for searching the category labels in the ground-truth, and the selected labels were finally assigned for the image patches. Then, 5,000 pixels were randomly selected as samples in the experiment. The ratio of the training and verification sets was 4:1. A 5-fold cross-validation method was adopted during the training process, and the models with the best performances of the validation set were saved. The numbers of the training and test samples in the three data sets are shown in Tables 5 to 7. During the training process, the number of samples per batch was 64. The maximum iteration batches were 50 epochs for both Dataset 1 and Dataset 2, and 100 epochs for Dataset 3. Then, an Adadelta optimization algorithm was adopted in the training process. The initial learning rate was set as 0.01. Finally, the trained network was used to classify the entire image.

TABLE 5 The Number of Samples in the Dataset 1

TABLE 6 The Number of Samples in the Dataset 2

TABLE 7 The Number of Samples in the Dataset 3

B. Evaluation Metrics

In the present study, in order to examine the performances of different methods, the experimental results were assessed by three evaluation metrics, namely the overall accuracy (OA), Kappa coefficient, and F1-measure. It was found that the OA could be calculated by the ratio between the number of correctly classified samples and the total number of samples. Therefore, the OA was defined in this study as follows:\begin{equation*} OA=\frac {M}{N}\tag{7}\end{equation*} where $M$ represents the number of correctly classified samples and $N$ is the number of total samples. The Kappa coefficients measure the consistency between the ground-truth data and the final classification results [62], which can be written as follows:\begin{equation*} Kappa=\frac {OA-P}{1-P},P=\frac {1}{N^{2}}\sum \limits _{i=1}^{C} {m_{i} k_{i}}\tag{8}\end{equation*} where $C$ is the number of categories; $m_{i}$ denotes the number of correct samples for the $i^{th}$ category; and $k_{i}$ indicates the number of samples of the $i^{th}$ category predicted by the classifier. The F1-measure is a harmonic mean between precision and the recall [63]. It is useful for imbalanced classes and is defined as shown in Eq. (9):\begin{equation*} F1-measure=2\times \frac {Precision\times Recall}{Precision+Recall}\tag{9}\end{equation*} In the equation, “precision” indicates how many of the samples detected by the classifier in each class were correctly classified, and “recall” represents the ratio of correctly classified pixels to the total number of samples in each class [63].

C. Comparison and Analysis of Results

1) Experimental Results of Dataset 1

Fig. 6 shows the accuracy and loss curves during the training process of Dataset 1. In the figure, the horizontal axis represents the number of epochs, and the vertical axis denotes the accuracy and loss values. In addition, for Dataset 1, the increases in precision and decreases of the loss values during the early stages of the network training were very fast. The network accuracy was approximately 98%, and the loss was approximately 0.05. The network had reached a stable state after the first few epochs. Subsequently, the trained network was used to classify the entire image. The results of the method which was adopted in this study, along with other comparative experimental results, are shown in Fig. 7.

FIGURE 6.

Accuracy and loss curves of Dataset 1.

FIGURE 7.

Marine oil spill detection classification results of Dataset 1.

Fig. 7 intuitively shows that this study’s proposed method had displayed high purity results in regard to visual effects. There were fewer spots on the surface of the sea, and the amount of seawater misclassified as oil spills had been greatly reduced when compared with the results of the other examined methods. As detailed in Table 8(where the brackets after the OA in the table show the variance of the classification accuracy), from a quantitative perspective, it was confirmed that the proposed method had achieved the highest results on all three indicators (OA, Kappa coefficients, and F1-measure).

TABLE 8 Classification Accuracy of Oil Spill Detection of Dataset 1. The Most Accurate Results are Indicated in Bold

When the traditional classification methods were used, it was found that regardless of whether T3 or PF was used as the input, there were generally only minimal differences observed in the obtained classification results. Furthermore, many spots were still visible on the sea surface and the overall accuracy was less than 98%. Also, the Kappa coefficients were all less than 0.9. In particular, the results of T3-SVM were observed to be the least accurate. It was found that the classification accuracy of the PFs as input in Dataset 1 was slightly lower than that when the T3 was the input. As mentioned above, traditional classification methods rely on the quality of feature extraction, which can potentially lead to different performance rates for the various feature combinations. However, when compared with directly using the original data for oil spill detection, the method using PFs as the input was based on the target electromagnetic scattering characteristics and the prior knowledge of the model. The performances of each of the classifiers tended to be stable, without many variations observed in the classification results. Nevertheless, due to the insufficient or redundant information covered by the different features or feature combinations, as well as other false identifications (such as the drilling platform in the upper right corner of the image), some objects were incorrectly identified as oil spills. In addition, the traditional methods had displayed poor edge recognition effects for the oil spills, as indicated by the red circle in Fig. 7.

It has been found that when compared with traditional methods, the CNN-based methods have the ability to more effectively improve detection performances. The spatial information is well presented and extracted through convolution and pooling operations. Therefore, it was clearly evident that the T3-CNN and PF-CNN methods significantly reduced the impacts of the sea speckle noise, which subsequently improved the powerful feature learning ability of the CNN. More importantly, the T3 had identified the most vital information via the CNN network. Therefore, the results of the T3-CNN were found to be slightly better than those of PF-CNN.

Furthermore, it was successfully confirmed that the method proposed in this study had shown the ability to accurately extract and fuse the two high-level features of the CNN. The method also adopted an SVM classifier to replace the weaker Softmax classifier in the CNN, which increased its ability to solve nonlinear problems. It was determined from the detection results that the proposed method’s performance was superior to the other examined methods, and the OA was further improved by approximately 1% when compared to the results obtained using the CNN-based method. In addition, when compared with the traditional methods, the OA, Kappa coefficients and F1-measure had been remarkably improved by the proposed method. This was particularly true for the Kappa coefficients. Also, using the results of this study’s comparison between the T3-SVM and the proposed method, the effectiveness of the feature extractions of the CNN method was further validated.

Table 9 shows the classification accuracy using different principal component combinations following the CNN feature extraction. It was observed that following the dimension reduction by PCA, the time used had been significantly reduced. Then, with the principal component increases, the classification time also significantly increased. Thereby, while considering the trade-off between the time cost and the accuracy, the first three principal components were selected as the fusion for the classification process in this study.

TABLE 9 Experimental Results on the Importance of the PCA of Dataset 1

Fig. 8 presents the different performances among the T3-CNN-SVM, T3-CNN-ML, T3-CNN-NN, and T3-CNN-KNN (K-Nearest Neighbor algorithm) in this study’s experiments. During the experimental processes, K was set as 7, and different classic classifiers were used to replace the Softmax of the CNN. The results demonstrated that the RBF-SVM based method had displayed the highest classification accuracy. Meanwhile, when compared with the traditional classification results of the T3-SVM, T3-ML, and T3-NN (Table 8), it was observed that the classification effects of the T3-CNN-SVM, T3-CNN-ML, and T3-CNN-NN had been notably improved. These results indicated that the classification effects could potentially be improved by using the features extracted from the CNN, rather than by directly using the original data or features. Moreover, the CNN-based feature mining processes could stably enhance the different classifiers in order to reach higher accuracy levels, as well as reducing the performance differences between each type of classifier.

FIGURE 8.

The classification results obtained by using different classifiers after CNN feature extraction and fusion of Dataset 1: (a)T3-CNN+SVM; (b)T3-CNN+ML; (c)T3-CNN+NN; (d)T3-CNN+KNN.

2) Experimental Results of Dataset 2

Fig. 9 shows the accuracy and loss curves during the training process of Dataset 2. It can be seen in the figure that a small fluctuation had occurred in the pre-training period. However, after 20 epochs, the curves had basically converged. The classification results and detection accuracy of Dataset 2 are shown in Fig. 10 and Table 10, respectively.

TABLE 10 Classification Accuracy of Oil Spill Detection of Dataset 2. The Most Accurate Results are Indicated in Bold

FIGURE 9.

Accuracy and loss curves of the Dataset 2. Note: In the figure, train_loss/acc indicates the accuracy and loss curves on the training samples; and val_loss/acc indicates the accuracy and loss curves on the validation samples.

FIGURE 10.

Marine oil spill detection classification results of Dataset 2.

The oil spill observed in Dataset 2 was generally presented as a slender type oil spill. As can be seen from the classification results in Fig. 10, this study’s proposed method still achieved the best results on this dataset, with OA, Kappa coefficients, and F1-measure of 98.89%, 0.8814, and 0.9423, respectively. It was also observed that the traditional methods had displayed a large number of speckles on the sea surface, especially in T3-ML. Although it was found that the T3-NN method performed well on the sea surface, it had displayed poor preserving abilities on the overall shapes of the oil spills, particularly in regard to the edges of the oil spills. It should be noted that the PFs were built on the scattering characteristics of the objects, which involved both stable and robust information leading to the stable performances of each classifier [18], [23].

In regard to the CNN-based method, it was observed that there were also some speckles on the sea surface, which may have been caused by the failure to effectively use a variety of high-level features and the insufficient performance of the top classifier. However, the edges of the oil spills remained relatively complete when compared to the results obtained using the classic method. In contrast, the method proposed in this study had effectively reduced the sea speckles. In addition, there were no false identifications as had appeared in the results of the other examined methods, which indicated that the fusion features in the CNN had displayed strong discrimination and representativeness abilities. Also, in accordance with the quantitative accuracy evaluation shown in Table 10, the accuracy of the proposed method was determined to be 98.89%, with 1.3% increment compared to that of the T3-CNN. Meanwhile, the Kappa coefficient was also significantly improved to 0.8814. Therefore, as proven by the results achieved using the method proposed in this study, its performance was found to be stable and superior to the other examined methods.

Table 11 details the classification accuracy and time costs using different principal component combinations. Similar regulation results as Dataset 1 were observed. Therefore, in the present study, the first three principal components were recommended as the fusion features.

TABLE 11 Experimental Results on the Importance of the PCA of Dataset 2. The Most Accurate Results are Indicated in Bold

Fig. 11 provides the classification results obtained using the different classifiers following the CNN feature extraction and fusion of Dataset 2. It can be seen in the figure that the method based on RBF-SVM had once again achieved the highest classification accuracy.

FIGURE 11.

The classification results obtained by using different classifiers after CNN feature extraction and fusion of Dataset 2: (a)T3-CNN+SVM; (b)T3-CNN+ML; (c)T3-CNN+NN; (d)T3-CNN+KNN.

3) Experimental Results of Dataset 3

The accuracy and loss curves during the training processes of Dataset 3 are shown in Fig. 12. There were more types of bio oil film in Dataset 3 than in the first two datasets. Therefore, there were certain differences between the training set and the test set curves. This may be related to the partitioning of the sample data, resulting in the model’s performance on the training set is slightly better than the validation set. However, the overall trend was consistent. It was found that as the epoch increased, both the accuracy and the loss curves tended to be stable. Fig. 13 and Table 12 detail the classification results and detection accuracy of Dataset 3, respectively.

TABLE 12 Classification Accuracy of Oil Spill Detection of Dataset 3. The Most Accurate Results are Indicated in Bold

FIGURE 12.

Accuracy and loss curves of the Dataset 3. Note: In the figure, train_loss/acc indicates the accuracy and loss curves on the training samples; and val_loss/acc indicates the accuracy and loss curves on the validation samples.

FIGURE 13.

Marine oil spill detection classification results of Dataset 3.

Dataset 3 contained three substances released by humans: crude oil, emulsified oil, and plant oil. In the current study, plant oil was used to simulate a natural biogenic slick. This experiment was mainly used to verify the ability of the proposed method to distinguish oil spills from biogenic slick. Biogenic slick refers to the natural surface slick formed by organic substances secreted by marine fish and some types algae plants [59]. It also is indicated by dark spots on SAR images, which makes it difficult to distinguish an oil spill from biogenic slick using traditional SAR image classification methods. Fig. 13 and Table 12 show the results and accuracy of the different methods, respectively. Once again, it was found that the proposed method had achieved the best performance.

For the three traditional methods, when the T3 matrix was directly used as an input, the oil spills could not be distinguished from the biological oil slicks. It was observed that almost all the oil film in the SAR images had been classified as oil spills. Meanwhile, a large number of misclassifications appeared on the sea surface, and the detection results were generally very poor. When the PF was used as the input, the three methods (PF-ML, PF-SVM, and PF-NN) were found to effectively distinguish the oil spills from the biological oil slicks, and the effects were noticeably improved. However, these Pol-SAR features possessed poor noise resistance, which tended to still induce many misclassifications of the sea surface objects. The T3-CNN and PF-CNN had extracted the spatial information of the PolSAR images and reduced the false identification rates of the sea surface objects. However, it was clear that the proposed method had the best detection results, including the highest OA, Kappa coefficients, and F1-measure of 97.56%, 0.7795, and 0.8005, respectively. When compared with the other methods examined in this study, the newly proposed method had not only significantly reduced the speckles on the sea surface, but had also shown the ability to accurately distinguish the biogenic slicks from the oil spills. In addition, the proposed method had also reduced the misclassification phenomena associated with the biogenic slicks. The detailed result information is shown in Fig. 15.

FIGURE 14.

The classification results obtained by using different classifiers after CNN feature extraction and fusion of Dataset 3: (a)T3-CNN+SVM; (b)T3-CNN+ML; (c)T3-CNN+NN; (d)T3-CNN+KNN.

FIGURE 15.

Details of marine oil spill detection results of Dataset 3: (a) PF-SVM; (b) T3-CNN; (c) Proposed method.

In addition, it should be noted that the edge area of the two oil films located in the middle and bottom right of the figure were wrongly divided into biological oil film. As mentioned in the previous related literature [17], the outermost parts of mineral oil spills may form thinner slicks over time. Therefore, the red zone around the edges of the emulsion and crude oil may be thinner film with properties similar to plant oil slicks. This could potentially explain the “misclassification” phenomena observed in Fig. 13. In future research, the addition of a super-pixel segmentation method, the region-based CNN [65] or conditional random fields (CRF) [66] should be considered. Also, the multi-scale spatial information of the oil spill edge regions should be thoroughly investigated in order to optimize the classification results [71].

The results of using different combinations of principal components for Dataset 3, along with the classification results obtained using the various classifiers following CNN feature extraction and fusion, are shown in Table 13 and Fig. 14, respectively. It was found that, similar to Datasets 1 and 2, the usage of the first three principal component combinations and the SVM classifier had achieved the recommended results.

TABLE 13 Experimental Results on the Importance of the PCA of Dataset 3

SECTION IV. Discussion

A. The Visualization of the Deep Features

According to the forementioned experimental results, the proposed approach shows great advantages in PolSAR oil spill classification. In this section, we visualize the deep features and discuss their superiority. After training the model, the feature centered at the pixel $(i,j)$ of a $15\times 15\times 9$ image patch is extracted through the trained CNN model. The size of the feature map after the third convolution layer is $2\times 2\times 20$ . It is stretched to a one-dimensional vector, i.e. a vector with a shape of $1\times 80$ , and recorded as $fea\_{}1$ . Similarly, the features of the fully connected layer in the network are extracted and recorded as $fea\_{}2$ , which is a vector of shape $1\times 240$ . These two features are regarded as high-level information regarding the center pixel and its surrounding neighborhoods. Then, two new matrices are constructed, and the two features are placed on the corresponding positions of $(i,j)$ , respectively. By traversing the entire graph, the feature images of the entire graph can be obtained. Then, PCA is used to reduce the dimension of features. Due to space limitations, this paper only includes the feature image of the first principal component after dimensionality reduction by PCA of Dataset 1, as shown in Figure 16.

FIGURE 16.

First principal component graph of deep features after PCA dimensionality reduction. (a) First principal component of fea_1; (b) First principal component of fea_2.

As can be seen from Figure 10, the deep features obtain a strong identification between the oil spill and sea water with no other false alarms. Compared with the polarimetric features extracted in Figure 5, the deep features exhibit a superior visual effect, because it takes the neighborhood information into account and reduces the speckle noise. In addition, most of the polarimetric features have similar responses to the oil spill and drilling platform in the right top in Figure 5, which may be the cause of the false alarms produced by the PF-SVM method. However, no such phenomenon has been found in the deep features, which proves that the two deep features extracted by the CNN have good discrimination and robustness, thereby proving the effectiveness of this method.

B. Discussion on Detection Rate and False Alarm Rate

In oil spill detection, the oil film is often confused with look-alikes, and affected by the speckle noise of polarimetric SAR imaging. Which often leads to more false alarms on the sea surface. Therefore, it is necessary to consider the accuracy of the detection results and false alarms rate (FAR). The FAR can reflect the proportion of negative examples among the samples classified by the classifier as positive examples. The results are discussed in further detail here. The results of accuracy and FAR for each category of Datasets 1 to 3 are shown in Tables 14 to 16.

TABLE 14 The Results of Accuracy and False Alarm Rate for Each Class in Dataset 1. The Most Accurate Results are Indicated in Bold

TABLE 15 The Results of Accuracy and False Alarm Rate for Each Class in Dataset 2.The Most Accurate Results are Indicated in Bold

TABLE 16 The Results of Accuracy and False Alarm Rate for Each Class in Dataset 3. The Most Accurate Results are Indicated in Bold

There are only two types in datasets 1 and 2, namely seawater and oil spills. For dataset 1, we can see that the most accurate detection result for oil spill is T3-SVM, but it introduces a lot of false alarms, and the FAR is as high as 70.65%. The extreme case of this phenomenon is equivalent to dividing an entire image into oil spills. At this time, the detection rate is 100% for oil spills, but it is conceivable that the FAR will also increase significantly, which can not be used in actual work. The method proposed in this paper can detect the oil spill well and reduce the false alarm of the sea surface greatly. It achieves a very good balance between the detection rate of oil spill and the FAR.

As can be seen from the results of dataset 2, compared with the classical method, the method based on CNN greatly improves the accuracy of oil spill detection, and at the same time reduces false alarms on the sea surface. The method in this paper further reduced the false alarm on the sea surface while maintaining the detection rate, and achieved the lowest false alarm rate. At the same time, it achieved the best accuracy on other indicators.

Dataset 3 contains three classes, therefore, the false alarm of biological oil slicks is also considered. The results are shown in Table 16. The classical method based on T3 matrix cannot distinguish between biological oil film and oil film, and a large number of false alarms are introduced in the sea surface, the FAR of oil spill and biological oil slicks is higher than 80%. PF-ML, PF-SVM, and PF-NN can distinguish three types of substances, but there are more false alarms of biological oil slicks on the sea surface. Compared with the classical method, the proposed method greatly reduces the sea false alarm. Compared with CNN method, the proposed method has the best accuracy in other indicators while maintaining the detection rate, and also reached the lowest FAR.

In general, through the above analysis, the proposed method in this paper can achieve a good compromise between detection rate and FAR. At the same time, the performance of OA, kappa, F1-measure and other indicators are better than other methods.

SECTION V. Conclusion

In view of the drawbacks of traditional offshore manually-driven oil spill detection methods with insufficient or redundant features, we proposed an oil spill detection method based on the fusion of deep learning features. The main contributions of this paper are summarized as follows:

1. A deep CNN suitable for oil spill detection is constructed to automatically extract identifiable deep features from the PolSAR data. In view of the problem that general deep learning methods cannot fuse multi-layer deep features, we extract two high-level features of the CNN network, then reduce the dimension of the two high-level features by means PCA, and finally fuse the features so as to include abundant target feature information. By comparing the visualization of high-level features with the traditional polarimetric features, we show that the features extracted automatically by this method exhibit superior representativeness and robustness.

2. The original Softmax classifier of the CNN network is replaced by the robust SVM classifier with an RBF kernel, which enhances the robustness of the classifier and the ability to solve non-linear problems. Therefore, we further improve the classification performance and enhance the ability to identify oil spills.

3. The oil spill and biogenic slick detection experiments carried out in this paper prove that, compared with other methods, this method can effectively reduce the false alarm rate caused by sea clutter or biogenic slick, and significantly improve the accuracy of oil spill detection.

However, in practical applications, due to the complexity of the sea surface environment, there are still unknowns in using deep learning for oil spills detection under various sea conditions. The proposed method is pixel-based and its calculation efficiency is not high. In future work, other deep learning frameworks should also be considered, such as FCN [60], Segnet [61], Resnet [50], and so on. In addition, research on feature level fusion and decision level fusion among different models can be carried out to further improve the accuracy and generalization performance of the models. In conclusion, the proposed method and experimental results confirm that the deep learning network has strong application potential in PolSAR oil spill detection. This study provides a reference for using deep learning for PolSAR oil spill detection in the future.