Introduction
A marine oil leak, which is generally caused by a severe shipwreck or drilling incident at an oil production platform, is highly destructive to the ecological environment. Once a leak accident happens, petroleum rapidly diffuses and slicks on the marine surface. The slicked oil film not only evaporates and puts toxic chemicals, such as hydrocarbons, into the air but also penetrates marine life [1]. The natural evaporation and biodegradation of the leaked oil are lengthy processes that are sensitive to the ambient temperature, offshore terrain, etc. Therefore, detecting the leak source quickly to reduce the amount of leaked oil rather than relying solely on cleanup measures after a larger volume has contaminated the ocean is a significant issue in marine oil leak hazard research. Remote-sensing techniques, with characteristics of high-resolution large-scale observations and steady patrol time intervals, can effectively tackle oil leaking hazards. It can observe the widely influenced region with short time and facilitate to detect the oil leak source [2].
An oil leak accident and corresponding detection process are shown in Fig. 1. Crude oil transmission pipelines or reservoirs can become damaged during the exploitation or transmission of offshore oil, which can cause crude oil to leak on the ocean floor. After leaking, petroleum particles will gradually rise, float, and accumulate. Oil particles can diffuse and drift under the force of waves and winds in a short time. When the crude oil emerges into a vast area of oil film on the sea surface, remote-sensing starts to exert its advantage in wild region monitoring. Remote-sensing continuously monitors the offshore oil spill area and captures the real-time oil film status. A large amount of remote-sensing data is then transmitted from satellites to the data center for storage, and this data can facilitate the leaking source detection.
With the rapid development of remote-sensing techniques, oil leak extraction using remote-sensing data has played an important role in oil spill research. Synthetic aperture radar (SAR) is widely utilized in remote sensing to collect data for oil detection. Singha et al. [3] proposed two artificial neural networks for the oil spill detection problem by image segmentation, feature extraction, and classification on SAR data. David et al. [4] utilized an automatic oil spill detection system that adaptively selected artificial neural networks or decision trees to classify SAR data. With the contributions of these researches, the oil film detection accuracy on remote sensing data has achieved significant progress. Under these circumstances, applying the promising technology to timely decision-making support response is urgent [5].
Numerical simulation models have provided general and practical measurements in the last decade while reproducing the oil leak process [6]. Targeting to tackle the oil spill trajectory forecasting, time estimating, and state assessment for arriving at specific areas of interest, the Eulerian method, and the Lagrangian method are the two mainstream solutions of the oil spill method. The Eulerian method involves the mass and momentum conservation equations or diffusion equations to simulate the continuum particle phase. Sarhadi Zadeh et al. [7] proposed a two-dimension hydrodynamic Eulerian model based on the Reynolds-averaged Navier–Stokes (RANS) equations. Ivorra et al. [8] designed second-order schemes for advection to reduce the numerical diffusion problem in the Eulerian model. The Lagrangian method, different from the Eulerian method, simulates the pathway of each particle as a discrete phase. Wang et al. [9] proposed a Lagrangian discrete particle algorithm based on the Princeton ocean model (POM) for oil spill transport simulation. Zelenke et al. [10] proposed the general NOAA operational modeling environment (GNOME), which applied a Lagrangian transport or trajectory model, to mitigate or avoid future damage to valuable natural resources caused by marine pollution. High temporal and spatial resolution marine data, including wind field, salinity, temperature, etc., are fed into these state-of-the-art approaches to simulate oil particle movement in the marine leak region. The simulation accuracy, to a great extent, relies on precise input data. However, before embracing a marine remote-sensing technique, marine data were generally acquired by the meteorological station and the tidal station [9], [11]. The number and location of stations limit the resolution of data. Though different spatial and temporal interpolation methods are devoted to exploring the pattern in historical data [12], [13], traditional numerical models are still delicate when this is the case. A credible simulation model based on limited historical data has become the research focus for numerical simulation models. Furthermore, numerical simulation methods suffer a large computation overhead and generally take hours to days to accomplish a simulation task [9], [14]. Balancing the efficiency and the accuracy by utilizing the numerical simulation method in the oil leak source detection guarantees the hazard in time decision-making feedback. A spill source detection method with high efficiency and high accuracy is urgently needed for oil spill accident pollution prevention.
In order to address the challenges mentioned above, an oil leak digital twin framework is designed to reproduce the practical accident. The digital twin is the seamless connector that integrates the physical and cyber space [15]. A variety of data is gathered and exchanged as the pipeline to empower the virtual twin to characterize the physical phenomenon. The virtual twin reflects the properties of physical accident and, furthermore, with the help of the optimization method, provides decision-making suggestions for the accident. Under the oil leak digital twin framework, a Monte Carlo-based deep Q-transfer-learning network (DQTN) for offshore oil leak detection (OLD) approach is proposed in this article. In this approach, the remote-sensing technique is utilized for oil leak region segmentation on an advanced synthetic aperture radar (ASAR). The lookup table technique is designed as an oil leak accident dictionary. Through querying the lookup table, the source detection range is settled in a small region. Then, a Monte Carlo-based estuarine and coastal ocean model (ECOM) oil simulation is developed to replicate the oil spill process. After that, the DQTN guides the detection of the oil spill source location. DQN is sufficiently trained in designed maze cases. Knowledge is inductively transferred to tackle the oil leak source detection task. The uncertainty in simulation is considered by the Monte Carlo method. The efficiency and accuracy are guaranteed by the lookup table and the DQTN. The contributions of the article are summarized as follows.
A DQTN OLD digital twin approach is proposed to locate an oil leak source effectively. Its high efficiency can facilitate understanding of oil leak-induced marine pollution.
The Monte Carlo technique is utilized to address the uncertainty in the ECOM parameters. The limited historical numerical model inputs are enriched by the statistical method.
A maze case is designed to pretrain the DQN, and the trained knowledge is transferred to the oil leak detection task, which significantly increases the efficiency of the proposed method.
The proposed method was verified by a real oil leak case from the Bohai Sea, China. The average accuracy was up to 97.54% and only took 184.5 min for the lookup table-based DQTN method to detect the oil leak source.
The rest of this article is organized as follows. Section II presents the related works about numerical simulation. Section III presents the proposed Monte Carlo-based oil leak source detection method in detail. In Section IV, the experiments conducted to evaluate the effectiveness of the proposed method are described. Finally, Section V concludes this article.
Related Work
Studies on marine oil leak events focus on oil film segmentation and oil diffusion simulation. In the following, state of the art on research and application of the two techniques are introduced. Section II-A addresses segmentation and extraction in remote sensing. Section II-B covers numerical simulation methods. In Section II-B, the numerical simulation model utilized in the proposed method is elaborated in detail.
A. Segmentation and Extraction in Remote Sensing
A marine petroleum leak accident is a destructive event with a wide-influenced and random location. When an accidental petroleum leak happens, timely emerging pollution prevention measures are required to avoid or reduce the damage to the marine environment. Accurate oil leak monitoring can provide valuable information for leak source detection. Traditional oil leak detection methods are primarily based on manual monitoring. However, under certain circumstances, the oil leak region is difficult to observe due to a wide range of leaked oil regions and the profound depth of the underwater source of the leak. This makes manual monitoring ineffective. With the help of the remote-sensing technique, these obstacles to monitoring can be gradually overcome. The long-term, large-scale continuous monitoring of specific areas by remote-sensing satellites can efficiently obtain information about oil leak accidents. This information includes the oil spill area, the oil type, and the oil film thickness. Guo et al. [16] analyzed the correlation between the scattering section and the related parameters of oil spill through an electromagnetic-scattering numerical model. They also built a model of oil spills to improve the accuracy of the oil spill monitoring. Some scholars have established marine oil spill extraction methods by multiple technologies. Shi et al. [17] proposed a method of integrating satellite remote-sensing, aviation remote-sensing, shipborne sensors, and other auxiliary equipment to monitor the extent of pollution and identify oil contaminants in a constructed large-scale emergency marine oil spill extraction system. Zhang et al. [18] constructed an oil spill feature database by studying the kinds of oil spills, their shape and scattering characteristics, and the textures of SAR images. They constructed an oil spill detection approach that combined the drone SAR and UV sensors to improve the efficiency of detection and extraction. From the perspective of marine oil spill spectral characteristics, Su et al. [19] made an observation that the spectral gap between the oil spill film and the seawater is greater than the variance of the seawater by mining the relationship between the optical remote-sensing satellite band and the spectral characteristics of the marine surface oil spill film. They extracted sea surface oil spill film based on this conclusion. Su et al. [20] used an SVM to extract features from optical remote-sensing images and established a spectral pattern sea surface oil spill monitoring model to find the position of the oil spill and detect the oil spill film. Sun et al. [21] considered the confusing phenomenon of oil film and seawater and used the spectral angle-matching algorithm to detect the oil film on the sea. By increasing the number of texture features, the accuracy of oil spill recognition can be significantly improved. Zou et al. [22] defined the confidence level of the oil spill remote-sensing information extraction based on the extraction indices of the marine oil spill incidents and the segmented maps for further spill identification.
State of the arts are dedicated to precisely extracting the oil film on the remote sensing data. The oil film information can help the implementation of the chemical petroleum degradation measure at the marine. However, the degradation measure is facing the challenge of petroleum pouring into seawater continually. Remote-sensing satellites have a long revisit time and high cost in monitoring. Extracting oil film based on low amounts of remote-sensing data cannot restore the whole leak process and, consequently, provide comprehensive decision-making support. This solution will waste enormous human and material resources. The leak source detection is more urgent to decrease the total volume of the leaked oil. Without the coordinates of the accident position under the water, it is challenging to decrease the oil film diffusion from the source. The numerical methods are designed for the oil leak accident simulation. Based on a series of kinetic equations, the numerical methods temps to retrieval the accident with the help of the oil film on the remote-sensing data.
B. Numerical Simulation Methods
Effectively determining the oil spill source can help postdisaster pollution prevention. Remote-sensing data are needed to collaborate with simulations and algorithms in order to determine a marine oil spill source [23]. A classical ocean numerical simulation model can accurately simulate the spread of oil spills on the sea. The accurate and efficient extraction of oil after obtaining marine oil spill data is a vital issue. In order to achieve better accuracy and efficiency in monitoring marine oil spills, existing studies propose continuously monitoring and detecting the diffusion of oil spills by combining numerical models with remote-sensing data. Liu et al. [24] proposed a method for predicting oil slick trajectory by combining oil slick satellite data with the Lagrange orbit model. Zodiatis et al. [25] proposed a MEDSLIK oil spill model to predict the diffusion of oil by analyzing remote-sensing images [26]. Later, the MEDSLIK oil spill model was improved in the MEDSLIK-II Lagrangian model, which combined SAR and optical image data to simulate the oil slick diffusion and transformation processes [27]. Based on remote-sensing data for marine oil spill monitoring and driven by dynamic remote-sensing data, Yan et al. [28] used a back propagation neural network to find oil spills. This method has certain limitations; the identification of an oil spill's location depends on experience or related accurate news reports in the initial simulation of the source. Furthermore, this method cannot adjust and model an actual oil spill for extended time intervals. Chen et al. [29] propose an oil leak detection algorithm based on cross-entropy, which combined remote-sensing data with an ocean oil spill model and ECOM.
Compared to other oil leak numerical simulation models, the ECOM is comprehensive and efficient. The ECOM considers diverse influence factors for oil particles to build dynamical equations that guarantee the simulation's accuracy. In addition, the ECOM is rigorous regarding the input data. Without high spatial and temporal resolution in the data, the model can still effectively simulate an oil spill. The ECOM is also a relatively efficient tool compared to other widely utilized numerical models, such as FVCOM. Thus, in this research, the ECOM was chosen to simulate the oil spill process.
The ECOM is a relatively mature 3-D hydrodynamic model suitable for shallow seas developed from the marine hydrodynamic POM [30], [31]. It has requirements for initial conditions, open boundaries, and grid settings. Its application involves complete thermodynamic equations. This article uses the 3-D oil-spreading module in the ECOM to simulate the trajectory of oil spills, including various processes, such as diffusion, retention, evaporation, and emulsification. The relevant parameters of the model design include oil type, oil density, overflow location, release depth, number of oil particles, tidal composition, wind field data, etc.
In the ECOM, the oil film is composed of a large number of oil particles. The oil particles leak into seawater at a specific rate at the point of incidence of the oil spill and then spread by advection. If
\begin{equation*}
\vec{V_{t}}= \vec{V_{a}} + \vec{V_{s}} \tag{1}
\end{equation*}
At each time interval
\begin{align*}
\Delta S &= \sum V_{t,k}\tau t_{k} \\
\sum \tau t_{k} &= \Delta t \tag{2}
\end{align*}
\begin{equation*}
\tau t_{k} \leq \left[ \frac{u_{k}}{\Delta x} + \frac{v_{k}}{\Delta y}\right]^{-1} \tag{3}
\end{equation*}
Oil particles will diffuse after convection and diffusion in each time interval. Diffusion is also an essential part of the early migration of oil particles. Due to evaporation, emulsion, etc., the quality of the oil particles gradually decreases. When oil particles reach the coast, they will be adsorbed on the coast or partially reenter the waters depending on the coastal conditions. After completing all the calculations, the convection, diffusion, evaporation, and emulsification processes of all the oil particles in a time interval have been completed. Only the temperature, wind, and flow field conditions need to be changed in the next time interval, and the entire calculation process is repeated. For more details, please see [31].
In this article, embracing the advantage of the digital twin framework, the proposed method is utilizing the numerical simulation method, the ECOM, for oil leak accident simulation and the intelligent method for leak source location searching. To dig into the inner connection between different simulation results, an intelligent method, the DQN method, is utilized to lead the agent to approach the oil leak source iteratively. However, there are some problems that obstacle the current method accurately and efficiency detecting leak sources as follows.
The uncertainty in the ECOM model requires to be evaluated. Even though the ECOM model has been applied to simulate several practical incidents, there are still some shortcomings that can influence the quality of the simulation. The rough standard in resolution for input parameters brings in some epistemic uncertainties. For example, the ECOM model assumes that the power and the direction of the wind for the whole research are all the same. The assumption is opposed to the objective principles. These uncertainties can be amplified by the numerical simulation algorithm and mislead the leak source detection.
Training DQN method requires network-tuning iteratively. The DQN method can tackle the more complicated problem than the traditional Q-learning method by utilizing the network to extract the insight pattern between states and actions. The ECOM is utilized to evaluate and guide the variables in the DQN to converge in each iteration. Because the numerical simulation is extremely time-consuming, accelerating training procedure is a significant issue needed to be tackled.
Oil leak source search region is extremely large. The primary principle for the DQN method is to manipulate the agent to approach the leak source in the region under the principle of the Q-network. The agent searches the leak source location by a series of actions, which guild the agent to walk in the search region. The trajectory of the agent connects from the initialized location to the target location. The length of the trajectory somehow depends on the original start location of the agents. The long search trajectory takes more execution time than the short one. Randomly initializing the start location of the agent makes the efficiency of the oil leak location search method volatile. Thus, how to steadily improve the efficiency of the DQN method is a significant problem that requires to be tackled.
To tackle the first issue, the Monte Carlo method is proposed to estimate the uncertainties of the input parameters, such as the historical wind field, through iteratively sampling. The Monte Carlo sampling method generates more potential input parameters instances and brings in more possible simulation results into consideration. To accelerate the training procedure of the DQN method, a pretrained problem is designed to reproduce the oil leak source detection problem. Before the oil leak accident, the network can be trained in advance and, when the accident happens, the trained knowledge can be transferred to tackle the leak source detection issue. For the third issue, the lookup-table technique is designed to simulate different oil leak scenarios in the target region. The efficient query lookup table procedure can significantly reduce the search area. Thus, the Monte Carlo technique, the DQTN, and the lookup table technique are proposed to tackle both the accuracy and the efficiency issues in the oil leak source detection.
Data and Proposed Method
The Monte Carlo-driven OLD method utilizes the ECOM oil leak simulation module and the DQTN method to optimize the oil leak source location. The Monte Carlo statistical technique is a stochastic method that is utilized to determine the uncertainties in the simulation. In the following, the study area in this research is first introduced in Section III-A. The entire workflow is then introduced in Section III-B. The four primary parts, which are the oil leak location extraction and ECOM simulation setup, the Monte Carlo oil leak evaluation, the oil leak lookup table design, and the DQTN optimization method are illustrated in the Section III-C–III-F, respectively.
A. Study Area and Remote-Sensing Data
A severe oil spill accident in the Penglai region of the Bohai Sea occurred on June 4, 2011, causing more than 5500 square meters (accounting for 7% of the total area of the Bohai Sea) to be affected by oil spill pollution. The accidental oil leak had a terrible impact on the surrounding aquatic industry in an area roughly defined by lat
The accidental oil spill was monitored by the Earth observation mission, environmental satellite (Envisat), of the European Space Agency. Envisat was launched on March 1, 2002, and continues to orbits the Earth [32]. ASAR, which works in the
B. Monte Carlo-Based DQTN Offshore Oil Leak Detection Method
The overall workflow of the proposed method is shown in Fig. 2. In the proposed oil leak digital twin framework, data collected in the physical space are utilized to describe the oil leak accident in virtual space. The collected data can be classified into two types. The data collected from the Bohai sea environment includes the shorelines, water depth, tide, and climate data. The oil leak accident-related data includes the start leak time, leak last time, oil film area, and leak region. The numerical analysis in virtual space theorizes the oil leak source location information and provides the reference of leak source location for pollution prevention after an accident.
In the cyber space, to guarantee the high efficiency of the proposed method, a look-up table is designed to simulate different oil leak incidents in the Bohai Sea comprehensively. The marine surface's oil film is first captured and sent to the database by remote sensing. After a series of preprocessing, including geometric correction, coordinate transformation, etc., the support vector machine (SVM) method extracts the oil leak region from the remote-sensing data. Note that the geographic coordinate system is transformed into WGS84, and the geometric precision correction and an enhanced Lee filter are applied for ASAR data process [28], [34], [35]. The oil film is then discretized to a large number of points based on the remote-sensing data. Based on the extracted oil film, the oil leak source search area can be determined into a small area by the look-up table. Before tackling the practical oil leaking data, a pretraining environment is designed based on the oil leak accident to train the DQN to enhance the search performance. A large number of parameters defined in the Q-learning network is trained in the pretraining environment. Then, the trained network of DQN is transferred for leak source detection. For the oil leak accident, starting at the random point as a candidate oil leak source. The location selection principle is based on the trained DQTN method. For each location, the leaking scenario is evaluated. Based on these evaluation results, the DQTN method can guide the candidate oil leak sources converging to the target of the source location. The oil leak evaluation components include the ECOM numerical simulation method and the statistical Monte Carlo technique. The ECOM method simulates the accidental oil leak in the ocean based on the current leak source. The Monte Carlo technique is utilized to handle uncertainties of the ECOM model. After iteratively searching for the supporting pollution measurements, the oil leak source location can be determined by the proposed method.
C. Marine Oil Leak Detection and Numerical Simulation Model Setup
In this work, to comprehensively evaluate an oil spill, historical marine data are collected and fed into the proposed method. ASAR data provided by Envisat is utilized to extract the oil film on the marine surface. Historical marine environmental data are fed into the numerical oil spill simulation model. The details of the data preprocessing are presented in this section.
As shown in Fig. 3, the petroleum leaked, rapidly diffused, and accumulated in the offshore region on June 11, 2011. The dense oil film has a strong pattern with massively less grayscale than the marine water region, as shown in Fig. 3(b). The oil region and marine water can be classified into two classes to abstract the oil film extraction problem. The target problem is transformed into a binary classification task. The supervised method, SVM, which is an effective classifier for complex and noisy remote-sensing data [36], is utilized to tackle the oil leak extraction problem. Typical pixels of oil film and water background on remote-sensing data are tagged manually. The tagged pixels are called regions of interests (ROIs). The SVM is trained on a small number of ROIs and determines the optimal hyperplane. A hyperplane, which sums up the diversity pattern of the types, is utilized to classify the remained untagged pixels. Note that the kernel trick, the radial basis function, is selected to map the pixels into a higher dimension for classification convenience. The radial basis function is formulated as
\begin{equation*}
R\lbrace X_{1}, X_{2}\rbrace = e^{-\frac{||X_{1} - X_{2}||^{2}}{\gamma ^{2}}} \tag{4}
\end{equation*}
Remote sensing captured by Envisat in the Bohai Sea on June 11, 2011. (a) Spilled oil accumulated on the marine surface in the shape of an irregular curve. (b) Three-dimensional remote sensing, whose z-label depends on the gray value of each pixel.
In addition to the remote-sensing data preprocessing and classification, the history marine data are also vital, as they are required to be fed into the ECOM marine numerical simulation model. Based on the principles described in Section II, putting up the dynamic analysis in the simulation region is the primary theoretical solution. The research region requires discretization into a number of grids. Based on it, the history marine data are distributed in each grid. Interpolation is utilized to tackle the resolution difference problem between the historical statistical data and the grids. Marine history data, including temperature, salinity, wind field, and current, are fed into the ECOM. In addition, to simulate the oil leak process, the oil leak source coordinates and the amount of oil leak petroleum are required to feed into the proposed model. After setting up the total simulated time length and simulated unit time interval, the ECOM can start to simulate the wave movements and oil particle trajectories in the marine environment. Note that the start time of the numeric simulation is started two days in advance. The force is initialized on the open boundary of the grids. Two-day “warming up” simulation transmits momentum and kinetic to the grid, which covers a vast area. The “warming up” procedure guarantees the dynamical state for the holding grid similar to the practical situation at the oil leak start time. This procedure can improve the oil particle trajectory estimation accuracy.
D. Monte Carlo Technique-Based Oil Leak Evaluation Method
The Monte Carlo technique is a statistical method to tackle the uncertainty in numerical simulation modeling. Marine numerical simulation models attempt to reproduce the historical status of the ocean. Because of the broad simulation region and relatively unreliable long-term buoy sensors employed, high resolutions and proper environment parameters are limited with historical ocean data. Incorrect input parameters significantly influence a simulation's effectiveness. From a statistical aspect, the Monte Carlo technique decreases the uncertainty in numerical simulation problems by massively sampling the input parameters. In this section, the Monte Carlo technique for the ECOM input parameters is introduced in detail.
The definition of uncertainty in a numerical simulation model is the primary obstacle to reducing uncertainty influences. Because of the limitation of the data acquisition sensors, deviations will exist in the data from the sensors. Sampling is the primary methodology of the proposed Monte Carlo technique. The Monte Carlo technique decreases the uncertainty by iteratively sampling and validating. Note that the sampling principle is based on the particulars of the issue. For instance, if the uncertainty is brought in by the measurement error of the input parameter, samples have to generate according to the precision of the data acquisition equipment.
\begin{equation*}
\hat{\phi }= E[\Lambda (P^{m}, \Delta D)] \tag{5}
\end{equation*}
Assume that the uncertainty is based on a normal distribution. The standard deviation of the uncertainty evaluation function is
\begin{equation*}
(P^{m}_{i}, \Delta D_{i}) = \frac{1}{\sqrt{2\pi }\sigma }\exp\left(-\frac{\left(x-P^{m}_{i}\right)^{2}}{2\sigma ^{2}}\right). \tag{6}
\end{equation*}
In this article, for the ECOM oil leak simulation, the uncertainty is predominantly brought in by the input parameters. The margin of the error influences the accuracy of the sensor measurement data. For an adequate sensor, the error is floated intolerance, and most errors by measurement are slight. The measurement data are the significant references, even though measurement error exists. The normal distribution is precisely suitable for the error estimation. Through the standard deviation, most generated data are close to the measurement data
E. Oil Leak Detection Lookup Table Design
The lookup table is a high-dimension matrix that is an effective reference for a similar problem. For a complex problem, the effect factors are diversity. The factors determine the look-up table's dimension and represent the index for retrieving value in the look-up table. The advantage of the look-up table is high decreases the execution time while tackling the same type of new coming issue. The look-up table is widely utilized in the computer-aided design domain [39]–[41]. Due to the oil leak detection target having significant demand on the efficiency, the look-up table technique is an effective method by preparing some instances for references in advance.
The setup and query procedure of the oil leak lookup table is shown in Fig. 4. For the oil leak practical accident, a diversity of parameters can influence the leak trajectory. The look-up table method can comprehensively take these factors into consideration. Note that the geographic information, such as coastline, bathymetric, etc., is entirely different for different regions, which has not come into the thinking of this research. For the specific research domain, the lookup table set up the grid, as introduced in Section III-D. Then, a large number of instances are simulated offline. The simulation results, which are the oil particles' coordinates in each time slot, are stored separately. According to the influence factors, which include the wind field
For the query procedure of the oil leak lookup table, the main target is to narrow the leak source search area. Except for the leak location, other influence factors are the index address to query the leak area in the target time slot. The principle of the fuzzy query is formulated as
\begin{align*}
&\Gamma = \omega _{t} * |T_{a} - T_{n}| + \omega _{s} * |S_{a} - S_{n}| + \omega _{\tau } * |\tau _{a} - \tau _{n}| \\
&+\omega _{w}*\left|\!\sqrt{(W_{a\tau }\!-\!W_{n\tau })^{2}+\cdots \!+\!(W_{a(\tau +m*\nu)}\!-\!W_{n(\tau +m*\nu)})^{2}}\!\right| \tag{7}
\end{align*}
\begin{align*}
&\mathop{\rm{argmax}}\limits _{j \in N^{j}} {\text{Acc}}_{j} \\
&s.t. \\
&\ \ \ \ \ \ \ \ \hat{T}, \hat{S}, \hat{W}, \hat{\tau } = \mathop{\rm{argmin}}\limits _{i \in N^{i}} \Gamma (C_{i}, C_{a})\\
&\ \ \ \ \ \ \ \ C_{a} = \lbrace T_{a}, S_{a}, \tau _{a}, W_{a}\rbrace \\
&\ \ \ \ \ \ \ \ C_{i} = \lbrace T_{i}, S_{i}, \tau _{i}, W_{i}\rbrace \\
&\ \ \ \ \ \ \ \ R_{j} = \Lambda (\hat{T}, \hat{S}, \hat{W}, \hat{\tau }, L_{j}),\ j \in N^{j} \\
&\ \ \ \ \ \ \ \ \text{Acc}_{j} = F_{mc}(R_{j}, RS_{a}),\ j \in N^{j}\tag{8}
\end{align*}
F. DQTN Oil Leak Detection Method
Reinforcement learning is a self-learning framework that assists an intelligent agent in tackling an established issue under a clear system of rewards and penalties. Intelligent agents iteratively attempt to achieve the goals under constraints. After extensive attempts, the agents have learned to tackle diverse issues effectively [42]. The DQN is a reinforcement learning method that utilizes deep learning as the decision component in the Q-learning method. The DQN has been targeted at addressing various problems in different fields in recent years, such as path-scheduling problems [43], [44], traffic flow control [45], [46], infrastructure distribution planning [47], [48], etc., and have achieved good results. The advantage of the DQN algorithm is that it only needs to define the starting state, ending goal, and problem rules (that is, the solution set space and reward and punishment mechanism), and it can perform repeated calculations spontaneously under the logic of reinforcement learning to achieve the expected end goal requirements [42]. Because of its automatic optimization capability, it can reach a solution even if the solution set space is large.
For the oil leak source detection problem, efficiency is the primary issue that obstacles the DQN technique tackling the problem. The DQN technique is utilized as a decision-making tool. It provides advice to adjust the location of the agent to approach the oil leak source. The DQN method builds the reward and punishment strategy to evaluate the simulation result. Although the possible solution set is large, the target is clear; the DQN algorithm can be applied to solve the current problem. However, the DQN technique requires sufficient iterations in the training process to optimize the neural network parameters. The DQN algorithm will mislead the agent direction, delaying the detection of the oil leak source. Therefore, the DQTN is proposed in this section to tackle the oil leak source detection issue.
The procedure of the DQTN oil leak detection method is shown in Fig. 5. A DQN is composed of an evaluation and target net. A pretraining environment instance is designed based on the oil leak source detection issues. The oil leak source detection is essentially a search problem that is similar to the maze problem. The DQN technique is pretrained by a large number of maze instances, and the target search experiences are transferred in the evaluation net to tackle the practical oil leak accident. The potential leak source coordinates and its corresponding information at state
The transfer learning technique is utilized to enhance the efficiency of the proposed method. Transfer learning is a technology to apply the knowledge learned in previous tasks to improve the solution for novel tasks [49]. Dynamic maze instances are designed to approach the oil leak accident issue as source tasks in transfer learning. Neither practical oil leak accidents nor numerical simulation methods are required in maze instances. The search ability of the agent is trained to guarantee the performance in the target domain, which is the oil leak source detection task. Though the source (maze) and the target (source detection) domains are similar, the tasks for these two domains are different due to the target domain requires numerical simulation to reproduce the accident. Furthermore, since both rewards for different states and actions are available, inductive transfer learning is utilized to transfer the searching knowledge. In each maze instance, the locations of the agent and target are randomly initialized. The input of the DQN technique includes the position of the agent, the decision space (that is, the range that can be moved at each step), the solution set space (all the movable ranges), the definition of the reward and punishment rules (in different positions). The output of the DQN technique is the score that the agent gains in each iteration. The DQN technique faces different maze instances in each iteration to enhance the complex practical oil leak source detection problem.
The overview architecture of the DQN algorithm is shown in Fig. 6. The DQN algorithm consists of two same fully connected neural networks and a replay memory pool. The DQN algorithm iteratively enhances its ability to manipulate the agent to make good action decisions by continually interacting with the environment. The evaluation network makes decisions under different circumstances in the environment. The architecture of the network is shown in Fig. 6(b). The environment evaluates the value of the action as a reward and feedbacks the updated state to the DQN algorithm. The replay memory pool, in Fig. 6(c), stores states, actions, rewards, and updated states sequentially for network optimization by the target network. Memories are loaded from the memory pool, and states and next states are fed into two networks to calculate the outputs, and Q value, respectively. The Bellman equation evaluates the temporal difference for Q values. The differences are utilized to optimize the evaluation network decision-making strategy.
Overview architecture of DQN. (a) Workflow of DQN. (b) Structure of the neural network. (c) Structure of replay memory.
Applying the DQN algorithm to the oil leak detection, the details of the Monte Carlo-based DQN OLD method is presented in Algorithm 1. The DQN algorithm starts with the initial memory unit
Considering the actual situation, the oil spill point is unknown, and the movement between the initially predicted oil spill point and the actual oil spill point is unknown. Therefore, both the state
The reward mechanics are different for dynamic maze instances and the practical oil leak case. A corresponding reward with a specific state
The DQN is optimized by cases in the memories. The parameters of the target network are loaded from the evaluation network. Random select groups of memories to, respectively, evaluate the Q values for the current state
\begin{equation*}
K_{i} = R_{i} + \gamma \times \text{argmax}_{a_{i}} Q_{e}\left(S_{i+1}\right) \tag{9}
\end{equation*}
\begin{equation*}
L = \left(Q_{a}(S_{i})|_{a_{i}} - K_{i}\right)^{2}. \tag{10}
\end{equation*}
The loss then is backpropagated to update the parameters in the evaluation network
Algorithm 1: Monte Carlo-Based DQN Offshore Oil Leak Detection Method.
Require:Initial marine environment parameters around the oil leak area.
Initialize the memory unit of DQN
Initialize evaluation network
Initialize target network
Initialize the start point for the oil leak source location
Set
Select action by
Update the candidate oil leak source location from
Simulate and evaluate by the Mento Carlo-based ECOM evaluation method to get the accuracy
Calculate the reward
Update memory unit
Randomly select memories from
Load parameters from the evaluation network
Calculate value of the historical decision-making knowledge value
Calculate losses between the historical knowledge and current knowledge
Utilize the root mean squared propagation method to decrease the loss
Update
Check the stop criteria according to the accuracy
Ensure:The computed oil leaking locations.
Experiments
Based on the major oil spill in Penglai, Bohai, in 2011, this experiment verified the accuracy and the efficiency of the Monte Carlo-based DQTN OLD Method. This section introduces the experiment information in detail, including the related data, the basic information of the platform for conducting the experiment, and the design of the experiments. Subsequently, the experimental results are analyzed and illustrated.
A. Experimental Setup
The experiment was conducted with the Python 3.5 and
According to the Bohai Sea oil spill accident in 2011, the ECOM was utilized to simulate the same oil spill process. The area of the Bohai Sea defined by longitude
The temperature during the accident was
For the look-up table, the oil leak source location and wind field data are taken into consideration. For the oil leak source location, 20 locations are chosen in the look-up table. 80, 85, 90, and 95 are selected in
Based on the architecture shown in Fig. 6, the structure of DQN is listed in Table II. States of size
For the oil leak detection process, the DQTN loads the pretrained model to help agents make action decisions in every state. Eight samples are generated by the Monte Carlo sampling evaluation method. In the simulation process, when the model ran to the time corresponding to the actual remote-sensing detection data, the simulation was stopped, and the current position of each oil particle in the simulation data was recorded to form a simulated oil film area. The current accuracy rate was the ratio of the oil samples extracted from the actual remote-sensing data in the simulated oil film area. For example, at iteration
B. Experimental Results
Leak source detection tasks are critical for an instant hazard prevention decision making [50]. Tackling oil leak source detection problems requires multidisciplinary knowledge, including the numerical model, the remote sensing technique, and the intelligent algorithm [51]–[53]. However, there is a great difference between the target, dataset, and methodology in these state of arts since the leak accident and leak monitor technique is different. Under these circumstances, these algorithms are not suitable for tackling current issues. Thus, this article evaluates the proposed Monte Carlo-based DQTN Offshore Oil Leak Detection (MC-DQTN-OLD) Method, commonly used methods, the particle swarm optimization (PSO), and the greedy method, which are also applied to the oil leak accident in the Bohai Sea. Targeting to verify the effectiveness of transfer learning in the proposed method, a comparison experiment is proposed to compare the DQTN method to the DQN method. We make the following observations.
The MC-DQTN-OLD method can effectively detect the oil leak source without prior knowledge. Fig. 7 shows a source detection instance for the proposed method. By querying the oil leak look-up table, the search region is reduced from 93 to 97 in the
direction and 48 to 52 in thex direction. The simulation oil pixels gradually approach the oil film on remote sensing data. After four steps, the accuracy of the candidate oil leak sourcey is up to 97.7%, which satisfies the stop criteria. Note that the grid, as a background in Fig. 7, is utilized in the ECOM method for oil leak simulation.[93, 52] The effectiveness of the proposed MC-DQTN-OLD method has been demonstrated by evaluating different instances with different start locations. In Table I, six instances are established to test the robustness of the method. Seeds for the build-in random function of Python and the random function of the external package, NumPy, are correspondingly settled. Because two different factors required random initialize, the start search location and the Monte Carlo wind field, two different random functions are utilized to control separately. The transfer learning is tested in this experiment. According to these six instances, the average accuracy of the DQN is 16.23%, and the execution time is 477.9 min. The average accuracy of the DQTN is 97.54%, and the average execution time is 184.5 min. The DQN only finds the DQN method one time in the six different cases, while the DQTN finds the leak source location in every case. The DQTN method search for oil leak source location by iteratively adjusting the agent's location. If the initial location is far away from the destination, the execution time will be affected. Since the target locations
or[93, 52] are detected, different instances' accuracy of six is close.[94, 52] The comparison algorithms, the greedy method, and the PSO method, cannot tackle the oil leak source detection issue. The greedy method moves the agent by distance. The “greedy agent” moves in the direction of decreasing distance. The agent can move up to 25 steps. The PSO method is a genetic algorithm that includes six individuals from a population in this research. Note that the stop criteria are the same for these three algorithms. As shown in Fig. 8, the greedy method can not find a proper oil leak location, and the accuracy is only 0%. The accuracy of the PSO method is 77.23%. Comparing to the PSO method, the proposed MC-DQTN-OLD method can achieve up to
improvement in accuracy.(97.54\% {-} 77.23\%) / 77.23\% = 26.30\% The proposed MC-DQTN-OLD method has a significant advantage in the algorithm efficiency when compared to the greedy method and the PSO method. As shown in Fig. 9, the execution time of the greedy method and the PSO method are 1229.68 and 903.95 min. Comparing to the greedy method, the proposed MC-DQTN-OLD method can achieve up to
improvement in efficiency. Compared to the PSO method, the proposed MC-DQTN-OLD method can achieve up to(1229.68 {-} 184.5) / 184.5=566.50\% improvement in efficiency. The proposed method significantly improves the efficiency of oil leak source detection while guaranteeing accuracy.(903.95 - 184.5) / 184.5=389.95\%
Accuracy comparison of the greedy method, the PSO method and the MC-DQTN-OLD method.
Conclusion
As a kind of pollution incident that seriously threatens the marine ecological environment, it is essential to take measures in time to prevent further damage. At this stage, research is focused on extracting the oil film area using marine remote sensing images to achieve monitoring effects. However, if the source of offshore oil spills is not located in time to reduce the total amount of crude oil leaked from the source, the scale of marine oil spill pollution cannot be effectively controlled. Therefore, in this study, based on remote-sensing images, the DQTN algorithm was used to study the problem of locating oil spill points in marine oil spills. The main conclusions are as follows.
This research used the digital twin architecture to tackle the oil leak monitoring and source detection issue. The DQTN algorithm is utilized to iteratively move the location of the oil spill point, used a marine oil spill model to simulate the spill, and evaluated the simulation results. This procedure is iterated until the exact oil leakage is obtained as a result, then this point was used as the location of the real oil leakage point. The DQTN algorithm makes full use of the results obtained from each oil leakage simulation in order to make the next decision. It can effectively use fewer resources in a more extensive solution set space to obtain the location of the oil spill point. In addition, the look-up table technique is utilized to improve the efficiency and the Monte Carlo method is applied to tackle the uncertainty in the ECOM simulation.
A major oil spill in the Penglai region of the Bohai Sea on June 2, 2011, caused huge damage to the marine environment and economic losses. Based on historical wind field data, a numerical simulation ECOM for marine oil leakage was constructed. Based on the remote-sensing imagery of June 11, the DQN algorithm was used to search for oil leaks at sea. After testing for different initial oil leaks, the average accuracy rate was 97.54%, the highest accuracy was 98.97%, and the average time was 184.5 min. The effectiveness of the proposed method was confirmed.
This study was based on remote-sensing images, used the proposed MC-DQTN-OLD, and evaluated the results of the oil spill simulation with the marine oil spill ECOM, which iteratively searched to achieve the purpose of locating the spill point of the oil spill event. The accurate depth of the oil spill point can, to a certain extent, assist staff to quickly find the actual oil leakage location, reduce the pollution of the environment from further oil leakage, and reduce the difficulty of environmental recovery after the disaster.