A Deep Learning Model for Remaining Useful Life Prediction of Aircraft Turbofan Engine on C-MAPSS Dataset

In the era of industry 4.0, safety, efficiency and reliability of industrial machinery is an elementary concern in trade sectors. The accurate remaining useful life (RUL) prediction of an equipment in due time allows us to effectively plan the maintenance operation and mitigate the downtime to raise the revenue of business. In the past decade, data driven based RUL prognostic methods had gained a lot of interest among the researchers. There exist various deep learning-based techniques which have been used for accurate RUL estimation. One of the widely used technique in this regard is the long short-term memory (LSTM) networks. To further improve the prediction accuracy of LSTM networks, this paper proposes a model in which effective pre-processing steps are combined with LSTM network. C-MAPSS turbofan engine degradation dataset released by NASA is used to validate the performance of the proposed model. One important factor in RUL predictions is to determine the starting point of the engine degradation. This work proposes an improved piecewise linear degradation model to determine the starting point of deterioration and assign the RUL target labels. The sensors data is pre-processed using the correlation analysis to choose only those sensors measurement which have a monotonous behavior with RUL, which is then filtered through a moving median filter. The updated RUL labels from the degradation model together with the pre-processed data are used to train a deep LSTM network. The deep neural network when combined with dimensionality reduction and piece-wise linear RUL function algorithms achieves improved performance on aircraft turbofan engine sensor dataset. We have tested our proposed model on all four sub-datasets in C-MAPSS and the results are then compared with the existing methods which utilizes the same dataset in their experimental work. It is concluded that our model yields improvement in RUL prediction and attains minimum root mean squared error and score function values.

SECTION I. Introduction

We are living in an era of industrial automation where our day-to-day activity depends heavily on a wide range of electrical and mechanical equipments varying from agriculture, process industry, power systems to the feild of transportation [1]. Every system requires maintenance operation at some point in time [2]. There are three types of maintenance techniques used in industries, reactive maintenance, preventive maintenance and predictive maintenance. In reactive maintenance repair operation is consider only when machine failure has occurred. In preventive maintenance the failure can be prevented by performing the time-based maintenance operation, while the predictive maintenance (PdM) lets you estimate the time-to-failure of an equipment for scheduling the maintenance tasks [3]. PdM in contrast to the other two approaches needs to acquire various machine parameters for condition based monitoring (CBM) of industrial equipment. This technique focuses on forecasting the error by modeling the degradation trends between input sensors and time-to-failure duration of the machine. So, the benefits of this maintenance strategy is that we can eliminate unplanned downtime, reduced maintenance costs and maximize the machine lifetime for safety critical circumstances. One such example is aircraft engine which requires continuous monitoring of the engine performance. The fault diagnostics and prognostics of aircraft engine has gained great attention over the last few decades [4], [5], [6], [7]. One important component in aircraft engine maintenance is to accurately determine its remaining useful life (RUL) for reducing the maintenance costs while attaining the reliability [8], [9]. RUL prediction model is developed based upon the degradation trends among the various condition monitoring sensors. This model helps in development of maintenance strategy in a targeted manner to eliminate unplanned downtime and maximize machine lifetime for safety critical circumstances. Early anomaly detection and timely warning of a failure is vital for maximum utilization of the system. There are basically three types of prognostics techniques used for estimating RUL, physical model-based approaches [10], [11], data-driven approaches [12], [13] and hybrid approaches [14].

Model based approach initially required a comprehensive understanding of the physical architecture of the machine and then applying the laws of physics to obtain the mathematical model of the machine for RUL estimation [15]. Mathematical models often take some simplifying assumptions with uncertainty management for a complex industrial machinery, which can impose serious limitations on these techniques and hence degrade the RUL prediction accuracy [16].

Data-driven based prognosis approaches use various statistical and machine leaning (ML) algorithms to discover the trends or patterns in the underlying sensor data to estimate RUL of the system. These techniques are suitable for complex industrial machinery and further, it does not require a thorough understanding of a complete engine or the process. Hybrid method combines both the physics and data-driven based model techniques [17].

In the past decade, data-driven based prognostics methods have been exploited by many researchers. These models estimate the RUL by analyzing the degradation trend and target trajectory of sensor data. Deep learning methods like autoencoder, convolutional neural networks (CNN), long short-term memory (LSTM) networks and their varianta and combinations have achieved a massive success in the fields of computer vision, speech recognition, video segmentation and predictive maintenance [18]. The major drawback of deep learning algorithm is that it requires a large volume of data for offline training and in the field of prognostics, it is very challenging to gather run time-to-failure sensor data especially for new machines. One way is that we can intentionally run a new system upto the failure mode but it is very prolonged, highly undesirable and expensive approach. Due to these limitations, researchers prefer some public datasets for the evaluation. In this work we have used commercial modular aeropropulsion system simulation (C-MAPSS) dataset which is basically a simulation of turbofan jet engine dataset provided by NASA prognostics center of excellence [19]. C-MAPSS dataset consist of four different multivariate time series units with different number of engines and each engine having different RUL. The dataset consists of twenty one sensors with three operating conditions with respect to time cycle for each engine.

In recent times, various work has been done on estimating the RUL of turbofan engine using deep learning methods such as CNN, LSTM along with their combinations and variants. LSTM networks have shown better results as compared to CNN based models [20], [21]. LSTM have shown excellent results because they are suitable for time-series data, they can learn the temporal features in multivariate system and minimize the root mean square error (RMSE) with respect to target predictions. In this paper, a LSTM based model has been proposed for RUL prediction of a turbofan engine. LSTM network can learn the association between target RUL values and sensor data but it alone cannot achieve state of the art performance due to various limitations like outliers, noise in the sensor values, un-normalized data and un-correlated sensor values. These shortcomings can reduce the performance of a LSTM network [22]. In this paper, we are focusing on implementing some preprocessing steps on the sensor data before it can be set as an input into the LSTM network. LSTM network when combined with effective pre-processing steps have the power to estimate the RUL with highly accuracy. These added steps involve correlation analysis, data filtering, normalization, and a modified piece linear degradation model for determining starting point of the degradation. It has been shown that the starting point of degradation which is also called the initial RUL has a great impact in determining accurate RUL predictions [23]. Our proposed modified piecewise linear degradation models help in efficiently calculating the starting point of degradation which in combination with the other pre-processing steps and LSTM network accurately predicts RUL for the given engines.

The main contributions of our work are enumerated as follows:

1. Novel piecewise linear degradation model for determining the starting point of engine degradation is proposed.

2. An LSTM network with effective pre-processing steps, i.e. correlation analysis with data normalization and moving median filter is proposed, which when augmented with the linear degradation model leads to an improved RUL prediction.

3. Hyperparameter for the proposed prediction model has been selected through iterative grid search based approach [24] to further improve the accuracy of our framework.

The organization of remaining paper is given as follow. Section 2 gives the detailed literature review on the existing methods on turbofan engine RUL estimation. Section 3 explain the dataset taken for experimental testing. Section 4 discusses the proposed methodology with comprehensive block diagram that elucidate the entire work flow. Section 5 discusses the experimental results. Section 6 is the conclusion.

SECTION II. Related Work

This section briefly reviews the existing literature on the turbofan engine RUL estimation. [25]. Traditional model-based techniques usually employ algorithms like Kalman filter (KF), extended Kalman filter (EKF) and particles filters to come up with mathematical formulation of machine based on multi sensor time series sequence data [26], [27], [28]. Classical degradation method such as Eyring model or Weibull distribution was implemented in [29]. Salahshoor et al. [30] used a unified framework of EKF based design for sensor data fusion algorithm to further enhanced the detection and diagnosis of degradation trends and system faults. Ordonez et al. [31] implemented the auto-regressive integrated moving average (ARIMA) model and support vector machine (SVR) methods collectively to estimate the RUL. The desired features can be created by analyzing prior learning about the degradation models as presented in [32]. In [33], it is suggested that failure thresholds or degradation state estimation is no longer required in learning-oriented approach. Khelif et al. [33] presented machine learning based support vector regression (SVR) model to project the direct association between multivariate sensor data or health index and the aircraft turbofan engine RUL.

Across all these techniques for turbofan engine RUL prediction, deep neural network-based methods have gained vast popularity. Zhang et al. [34] introduced a multi-objective evolutionary algorithm to expand and organized the deep belief network into multiple parallel networks simultaneously to accomplish the two convicting objectives i.e. diversity and accuracy. These networks attained a fine RUL prediction accuracy especially in case of complicated operations and in the presence of noise in input data [35], [36]. Saeidi et al. [37], proposed a naive Bayesian classification algorithm to measure the health index for turbofan engine. The pre-processing step takes the sensor data as input and apply moving average filter for removing the noise. It further categories the dataset into four different categories on the basis of time cycles i.e. time cycle values between 0 to 50 is labeled as urgent case which need immediate maintenance and further categorization is also done in a similar manner. Zheng et al. [38] proposed LSTM network combined with piece wise linear function for RUL for estimating the degradation trends. It achieves good results by applying piece wise linear function and data normalization. Wei et al. [39] proposed a Bi-LSTM network which can learn high level features in both direction and it can run training pass from forward to backward and backward to forward with back propagation algorithm. Wang et al. [40] proposed a hybrid network for turbofan engine in which trends and hidden pattern in long sequence sensor data is identified through LSTM network and short duration sequence was analyzed through time window method with gradient boosting regression (GBR). This method has two stages, offline stage to learn degradation pattern with LSTM network and TW-GBR used in online stage for extracting short sequence data. It also implemented standardization and sensor selection criteria. Babu et al [41], proposed deep CNN regression network for RUL estimation. The network consist of two dimensional convolutional layers for feature extraction followed by a fully connected regression layer for prediction. For 2-D convolution, first we have to convert our 1-D sensor data into 2-D with one dimension was taken as time and other taken as sensors amplitude. This model also extracts the spatial features very efficiently. Li et al [42], proposed deep convolutional neural network (CNN) for RUL estimation. The architecture of CNN is modelled in such a way that feature can be extracted from prepared 2-D sensor data by passing raw data into convolution layers, then flattened layer is added to convert extracted 2-D features into 1-D so that it can be given as an input to multilayer perceptron model with dropout layer for predicting RUL. Jayasinghe et al [43], proposed temporal convolution in which combination of CNN-LSTM network was used for turbofan engine dataset. The layers of the model have stacked by first applying data augmentation to create similar type of data for avoiding overfitting followed by data normalization which was then followed by 1-D convolution for feature extraction, lastly fully connected layer act as the bridge between output of 1-D convolution layer and input of LSTM layer. LSTM layer was then followed by fully connected layer for output prognosis prediction. Hong et al. [44], proposed a similar kind of network by stacking a 1-D convolution layer, residual layer, LSTM layer and a Bi-LSTM layer. Correlation analysis on sensor data for turbofan engine dataset was also performed. Mo et al. [45], proposed multi-head neural network for RUL prediction of turbofan engine. This network is different from the series network in such a way that they have implemented the parallel branches of CNN layer in series with LSTM network. Furthermore, fisher method in combination with recursive least squares and single exponential smoothing was also employed to find the prediction error and given it as an additional input into CNN-LSTM head for optimum performance. Zhao et al. [46], proposed an adjacnet neural network model for leanring the degradation pattern in a sensor data. The degaradation pattern mapping learns through morkov property i.e. estimating the next state of sequence with the assist of only present states.

Many researchers have used a piecewise linear degradation model in RUL prediction techniques. In this model, the starting point of the degradation is estimated often referred to as the initial RUL, many authors [47], [48], [49] have chosen its value on the basis of observations and no clear mechanism of selecting it has been proposed. Lan et al. [23], proposed an LSTM algorithm for RUL prediction, it presented a piece wise linear degradation model. The dataset is divided into time windows and difference between geometric distance (Euclidian) of each window is used to determine the value of initial RUL or the starting point of degradation. They have validated their model on only FD001 sub-unit of the dataset and selection of window size and thresholds are also not explained. Inspired by the results of this model, we have proposed an improved version of the algorithm, which is applied to the complete dataset and results in overall increased in the accuracy of our prediction model.

The existing methods presented so far showed impressive performance on RUL tracking; our proposed method further improves the performance of these networks − summarized in the results and discussion section. This is mainly due to the fact that we have incorporated and implemented effective data preprocessing steps into our prediction pipeline. In the previous works, pre-processing stages have not been completely exploited in order to find the right combination for enhancing the learning capability of deep learning algorithms. So, we propose one such combination along with the training of LSTM with a hyperparameter selected through grid search algorithm in order to achieve the optimum performance.

A. Problem Statement

The predictive maintenance techniques is different from the reactive and preventive maintenance techniques in terms of safety, reliability, efficiency and unnecessary downtime for aircraft turbofan engine [50]. These methods ensure reliable solution managing the health of turbofan engine to reduce the downtime, which leads to significant loss in revenue. Therefore, failures in a turbofan engines can cause catastrophic accidents due to its sensitive nature and it needs to be estimated prior in time so that we can provide maintenance services in order prevent any fatal incident.

In current scenario, due to the implementation of cyber-physical system (CPS, that link the cyber world with a physical world, called smart manufacturing), the industrial sector such as health care, nuclear power plant etc. generate enormous volume, velocity, veracity and variety of data. Therefore, with the rise of AI and availability of hardware computing resources, data driven based artificial intelligence (AI) predictive maintenance models have a capability to process big amount of real-world machines data with ease and predict heath index of aircraft turbofan engine in time before failure to prevent unwanted breakdown [51].

SECTION III. Description of C-MAPSS Turbofan Engine Simulation Dataset

C-MAPSS dataset released by NASA is developed in MATLAB environment as a tool for simulation of turbofan engines. C-MAPSS dataset was published in 2008 for $1^{st}$ International conference on PHM [19]. This dataset was published some time ago but still it has been actively used in research for evaluating the prognostics model with a focus on accurate estimation of RUL. This model have fourteen input parameters related to five rotating components of engine to simulate different degree of fault and deterioration of the model. A total of twenty one variables out of fifty eight different sensor responses is considered from the model for predicting the RUL. The three operating parameters of C-MAPSS simulation model are given in table-1 and the details of 21 sensors are given in table-2. The legends of last column in table-2 Trends indicates the degradation pattern of sensor data with respect to time, where “~ ” represents irregular sensor behavior, ${I}$ represent the parameter increasing with time and lastly, ${D}$ is the variation of parameter that decreases with time.

TABLE 1 Operating Parameters of C-MAPSS

TABLE 2 Output Parameters C-MAPSS Turbofan Engine Datasets

The main components of turbofan engine include nozzle, low pressure turbine (LPT), high pressure turbine (HPT), fan, low pressure chamber (LPH) and high pressure chamber (HPC). There are total of fourteen editable input parameters such as fuel flow, HPC efficiency modifier, LPT efficiency modifier etc. that allows you to simulate various operating behaviors. C-MAPSS sensor trajectories are further divided into four different units namely FD001, FD002, FD003, and FD004 corresponding to different operating conditions and fault modes. This dataset contains 709 engines for the training and 707 engines for testing which are of same type but with distinct manufacturing variation and initial wear, unknown to the researcher. The description of four sub dataset units with train and test trajectories and other details are given in table-3. In the start, all the engines in each sub-dataset are operating normally as seen from sensor behavior and originate the fault sometime later in their life cycle. In training sequence, complete run-to-failure data is available with a specified RUL labels as faults grows in the system and in test time degradation values are given up to some time prior to engine failure. Moreover, with different initial health conditions, there are distinct number of time cycle even for the same engine in dataset. The objective of this dataset is to predict remaining useful life cycle of engines in each sub-unit. The actual RUL label are given in the test data, which is used to validate the prediction results.

TABLE 3 Each Sub-Unit of C-MAPSS Dataset

It can be observed from table 3 that different sub-units of the main dataset have different running life cycle time e.g. in FD004 test data, the maximum life cycle time is 486 and minimum time 19.

SECTION IV. Proposed Methodology

This paper proposes LSTM based RUL prediction model for turbofan engines, which proves to be more robust than most of the existing models available in the literature. The increased robustness stems from the fact that we have integrated efficacious techniques from multiple alternatives and presented two types of improvements to be made in the LSTM based prediction model. The first type of improvement is the addition of more effective pre-processing steps, which in return puts a great influence on LSTM training. Secondly, enhancing the LSTM training procedure by a grid search approach for computing the effective hyper parameters. This framework manages to achieve higher prediction accuracy, measured in temrs of RMSE and score function values, than several existing works − see Section 5 for details. The framework of our proposed model is shown in Fig. 1. The specific steps of the prediction model are explained below.

FIGURE 1.

Proposed Framework for RUL predictions.

A. Correlation Analysis

First, we must prepare our health-to-failure data into an appropriate form for improving the accuracy of the LSTM network for effective training operation. C-MAPSS dataset consists of three operational settings and twenty one sensor signals of engines with machine life span length. These signals are then given as an input to correlation analysis method [31], [44], [52], [53] to discover the relevance of features with RUL. The algorithm excludes the sensor values which have a very little or zero correlation with RUL, this includes some parameter in engines that are basically controlled by a feedback controller and results into a near constant values or having an oscillatory behavior. These kinds of parameters do not play much part in RUL predictions. So, the selected feature signals are then given as an input into the data filtering stage. Statistical evaluation of turbofan engine degradation dataset gives us certain insight into the multivariate sensor data and furthermore reach towards the conclusion that whether a considered sensor is adequate for training the network or not. We can accomplish this abstraction by computing the value of correlation coefficient ’r’ which is a relationship between the sensors and RUL labels.

$$\begin{align*} r=&\frac {conv(x,y)} {s_{x}s_{y}} \tag{1}\\ r=&\frac {\sum _{i=1}^{n}(x_{i}-\bar x) (y_{i}-\bar y)}{\sqrt {\sum _{i=1}^{n}(x_{i}-\bar x)\left({\sum _{i=1}^{n}(y_{i}-\bar y)}\right)}} \tag{2}\end{align*}$$ where conv (x,y) is covariance between input sensor data (${x}$ ) and output RUL label (${y}$ ), $\bar {x}$ and $\bar {y}$ are the mean of input sensor data and RUL label, ${n}$ is the number of variables in a dataset and ${\mathit {S_{x}}}$ & ${\mathit {S_{y}}}$ are standard deviation of the two signals ${x}$ and ${y}$ respectively.

In [44] a correlation analysis is employed for dimensionality reduction to obtain the accurate results and to reduces the complexity of sensor data. This technique is primarily limited to FD001 sub-unit of C-MAPSS dataset, we have extended this approach to entire degradation dataset and comprehensively investigate the trends and irregular behaviors by analyzing the correlation matrix heat map [54] of C-MAPSS turbofan engine dataset. The correlation matrix heat map cells show the association of three operating settings and 21 sensors with output RUL labels as shown in Fig. 2. The correlation matrices are converted into percentage with dark green color representing higher correlation as opposed to light color which depict a low correlation value. The number of sensors selected from each sub-unit after the correlation process are given in table 4. After analyzing the correlation coefficient of sensor degradation dataset, we have concluded that 14 out 24 parameters in FD001 and 16 out of 24 parameters in FD003 reflect high strength of correlation with defined or monotonous behavior and we have omitted those variables that indicates less than 10% of correlation with the output RUL label. For the case of FD002 and FD004, correlation coefficient values in a heat map illustrated that these variables have little degree of association with output RUL labels and this correlation is more than 15% for best possible case. After analyzes the correlation matrix, un-correlated sensors are removed from the training process which are listed in table 5. So, only highly correlated sensor data are used for training the LSTM network which also increases the speed of learning for estimating RUL.

TABLE 4 Sensors Selection Based on Correlation Matrix

TABLE 5 Excluded Sensor List

FIGURE 2.

Correlation Matrix of C-MAPSS Dataset.

B. Filtering

The correlated data is passsed through a moving median filter for removing the outliers and noises in the sensors data. The choice of filter is made on its ability in removing the outlier while preserving the high and low frequency contents in sensor data and avoids any loss of data values. The time window size of moving median filter is adaptive and vary with respect to sensor values. The moving median filter belong to a type of non-linear digital filter, which is used to remove random unwanted noise especially when there is a high spike and short-term outlier present in the data points but preserving the high frequency information contents [55]. Median filter is used to identify such sensor values in turbofan engine which are considerably different from the other data points and then eliminate them. The algorithm of moving median filter is accomplished by sliding the window of appropriate length over the sensor data entries by entries and then replacing the corresponding values by computing the median of neighboring points specified in the window. Mathematically it can be expressed as:

$$\begin{equation*} {y_{i}^{j}}[n] = median({x_{i}^{j}}[n],\ldots,{x_{i}^{j}}[n+T])\tag{3}\end{equation*}$$ where ${n}$ = total number of sensor data points in each engine, ${i}$ = sensor variable, ${j}$ = engines, ${\mathit {x_{i}^{j}[n]}}$ denotes input data for sensor ${i}$ in engine ${j}$ and ${\mathit {y_{i}^{j}[n]}}$ returns the medians values across each sensor variables from FD001 to FD004 with a same dimension as of input data.

The algorithm of filter is accomplished by sliding the window of length ${T}$ over the neighboring elements in sensor data and computed the median for each considered window and median operation is then performed on these array vector to get the filtered output after applying ascending operation. So this processed data is then given to the next stages and hence put a significant impact on the output.

C. Data Normalization

The range of sensor output after analyzing from the graphs is from tens to thousands and if we use these raw values for training the network then accuracy will drop significantly [56]. The filtered signals are normalized to have same degree of range for efficient training of the network. Z-Score normalization [57] is used in this paper which first compute the mean ($\mu$ ) and standard deviation ($\sigma$ ) of each feature vector and then apply the following operation on each sensor output.

$$\begin{equation*} y_{i}^{n} = \frac {x_{i}^{c} - \mu ^{n}}{\sigma ^{n}}\tag{4}\end{equation*}$$ where, ${\mathit {y_{i}^{n}}}$ is the normalized value at $i^{th}$ time cycle for sensor ${n}$ , $\mathit {\mu ^{n}}$ is the mean value of all output of sensor ${n}$ , $\mathit {\sigma ^{n}}$ is the standard deviation of all ${n}$ sensor output.

D. Improved Piecewise Linear Degradation Model

It is observed that RUL is linear decreasing function with respect to time as the efficiency of the system degrades. However, as the system starts their operation, there is no degradation present in the sensor readings. This pre-processing step is basically implemented on output labelled data that takes input from previous correlation analysis stage and employs a piece wise linear degradation function on sensor values for finding the initial RUL or the starting point of degradation. All the labels till this deterioration point are constant out to the initial RUL value while the remaining RUL lables are represented as a linear line from that degrdation point up to zero life cycle time.

In this paper, we have presented an improved version of the automatic piece wise linear function [23] for output RUL labeling. This approach is self-governing that is sensitive on variation of the degradation trends and will automatically calculate the early point of sensor deterioration. The computation of initial RUL starts by dividing entire sensor time cycle with non-overlapping pattern into equal sized window length of ${w}$ and extract the sensor data from a given windows. We then calculate the centroid of each considered window by determining their mean values and geometric distance calculation is performed by subtracting the two subsequent windows to generate the trends in sensor data. As there are ${N}$ number of time cycles for given variable and window length of ${w}$ results into (g=N/w) geometric points for a given dataset. These geometric distances are computed using Euclidean distance method which is then squared and the degradation pattern from ${g}$ values is evidently detectable from the resulting plot as shown in Fig. 3. The centroid of window $w_{1}$ is first computed and subtracted from the other windows in a sequence to compute the variation in sensor values to reach on a point of deterioration based on the threshold value. The inflection point of the curve indicates the increase in sensor trends which is the initial RUL value.

FIGURE 3.

Degradation process represented in the form of windows of time cycle.

The proposed algorithm is given as Algorithm-1 is implemented for each engine. The minimum value of initial RUL among all the engines in a sub-unit is taken as the initial RUL for that sub-unit. Threshold level is dependent on the rate of rise in raw sensor data, its visual perception and how early we need to predict the faults in the machines for maintenance purpose.

In this paper, we have set different ranges of threshold (0.01 to 0.2) and window size (5,12) for calculating the initial RUL and validating the performance of our model. We have used different values of window sizes in order to compute the knee point in sensor data effectively. This choice stems from the fact that by further increasing the window size, the algorithm bypasses the knee point, which is the starting point of increasing/decreasing trend in sensor variables. Fig. 3 confirms that the trend in sensor data changes within few cycles, and if we further increase the window size then we cannot achieve true inflection point in these variations. As a result, initial RUL computation will not reflect the ideal knee point in the sensors’ behavior.

E. LSTM Model Development

The data from the normalization stage with updated RUL labels from the degradation model is used to train a deep LSTM network. Our proposed LSTM model consists of LSTM layers, dropout layers, fully connected layers and the regression layer. The output node of fully connected layer is a regression layer that gives the estimated RUL of turbofan engine. Our proposed model consists of four layers connected in a sequential manner with different number of hidden units and a dropout layer is also added in between the LSTM layers for enhancing the generalization of network to avoid over fitting. It is then followed by two fully connected layer with dropout layers and the final layer is the regression layer as shown in Fig. 1. Fig. 4 shows a basic LSTM cell that is essentially consist of three control gates: input gate, forget gate and output gate. The output of the cell is denoted by ${h_{t}}$ , which is a short-term memory sate in a network and ${C_{t}}$ is considered as a long-term cell state. The first gate in an LSTM cell is forgot gate ${f_{t}}$ , which is used to unlearn selective information stored in previous LSTM cell. The forget gate equation is given below.

$$\begin{equation*} f_{t}= \sigma (W_{f}.[h_{t-1}, x_{t}] + b_{f}) \tag{5}\end{equation*}$$ where $\sigma$ (), is called sigmoid activation function, which can control operation of forget gate. ${W_{f}}$ is the weight matrix, ${h_{t-1}}$ short term state from previous cell, ${x_{t}}$ is the input of cell, and ${b_{f}}$ is the bias vector of LSTM cell. The input gate controls the new information entering into the cell through following two equations: $$\begin{align*} i_{t}=&\sigma (W_{i}.[h _{t-1}, x_{t}] + b_{i}) \tag{6}\\ \bar {C_{t}}=&tanh (W_{c}.[h_{t-1}, x_{t}] + b_{c})\tag{7}\end{align*}$$ where $tanh = (e^{x}-e^{-x})/(e^{x}+e^{-x})$ . This value is calculated by the same short term state vector ${h_{t-1}}$ which is further used to update the new state of cell. ${W_{i}}$ and ${b_{i}}$ is the weight matrix and bias vector of an input gate respectively. The $\bar {C_{t}}$ computed from above equation is first filtered by it and then added to the long term state of the cell. ${W_{c}}$ and ${b_{c}}$ are the weight matrix and bias vector. After computing the value of forget gate (${f_{t}}$ ), input gate (${i_{t}}$ ) and ($\bar {C_{t}}$ ), long term state ${C_{t}}$ of LSTM cell is updated after applying given below matrix operation $$\begin{equation*} C_{t}= f_{t} \otimes C_{t-1}+ i_{t} \otimes \bar {C_{t}}\tag{8}\end{equation*}$$ where, $\otimes$ is basically element wise matrix multiplication operation between a specified variable and ${C_{t-1}}$ is the previous state of LSTM cell. Finally the output of LSTM cell is generated by the following two equation, $$\begin{align*} o_{t}=&\sigma (W_{o}.[h_{t-1}, x_{t}] + b_{o}) \tag{9}\\ h_{t}=&o_{t} \otimes tanh (C_{t})\tag{10}\end{align*}$$ The output state of LSTM cell ${h_{t}}$ , is obtained by filtering the output gate ${o_{t}}$ equation with a matrix of ${C_{t}}$ . ${W_{o}}$ and ${b_{o}}$ are the weight matrix and bias vector of the output equation. The values of the weight matrix and biases are computed by training the LSTM network.

FIGURE 4.

Structure of a LSTM cell.

F. Dropout Layer

The dropout layer is added to avoid the overfitting which inherently occur while training the deep neural network [58]. This regularization layer is added in between the fully connected layers and LSTM layers to increase the generalization of the whole algorithm to better track the predicted RUL with high accuracy. It drops some random portion of neuron according to the probability parameter defined in a network while remaining weights are trained by the backpropagation algorithm [59]. The expression for a dropout layer by choosing a linear activation function and considering the activity in unit ${i}$ at layer ${h}$ is given by the expression.

$$\begin{equation*} S_{i}^{h} (I)= \sum _{I < h} \sum _{j} w_{ij}^{hl} S_{j}^{l}\: \: with\: S_{j}^{0}=I_{j}\tag{11}\end{equation*}$$ where ${w}$ is the weight and ${I}$ is the input vector.

G. Fully Connected Layer

The fully connected layers gets the data from the final combination of LSTM and dropiut layer, so that the features extracted from LSTM layer are used to generate the output [60]. Due to its fully connected nature between all the neuron present in the network, it has a large amount of weight parameters which needs to be computed by training the network. Fully connected layer along with a dropout layer is followed by single regression layer for predicting the RUL. The mathematical calculation of fully connected layer is expressed below.

$$\begin{align*} H_{0}=&I \tag{12}\\ H_{l}=&\phi (h_{(l-1)} W_{fc}+b_{fc})\tag{13}\end{align*}$$ where ${I}$ is the input vector, $\phi$ is activation function of a neuron which is primarily a ReLu fucntion, ${h_{l-1}}$ is output from the previous layer, ${W_{fc}}$ is weight matrix of a specified fully connected layer, ${b_{fc}}$ is the bias vector and ${H_{l}}$ is output at ${l^{th}}$ layer.

H. Interpretation of Our Approach

We implement deep neural network combined with per-processing steps for efficient RUL prediction of turbofan engine. Furthermore, we have separately discussed the above-mentioned stages of prediction pipeline breifly in order to delineate a comprehensive understanding of our work. Therefore, to summarize our approach, we have presented a pseudo code of our proposed pipeline in Algorithm 2.

Algorithm 2 Proposed Methodology

1:

Inputs: Training Data,{${\delta _{T}}$ = (${x_{1}^{1}}$ ,${y_{1}^{1}}$ ),(${x_{2}^{1}}$ ,${y_{2}^{1}}$ )….(${x_{N}^{E}}$ ,${y_{N}^{E}}$ )}

2:

Parameters: ${N}~{\leftarrow }$ No. of sensor variables, ${E}~{\leftarrow }$ No. of Engines, ${\delta _{N}}~{\leftarrow }$ Filtered data, ${\delta _{C}}~{\leftarrow }$ Correlated sensors, ${M}~{\leftarrow }$ No of correlated sensors, ${\delta _{D}}~{\leftarrow }$ Normalized data, ${\delta _{P}}~{\leftarrow }$ Piece-wise linear RUL function, ${L_{N}}~{\leftarrow }$ No. of layers in LSTM, ${L_{D}}~{\leftarrow }$ Doprout layers, ${R_{L}}~{\leftarrow }$ Regression layer, ${L_{F}}~{\leftarrow }$ No. of neuron in FC layer, ${W}~{\leftarrow }$ Window Length, ${\alpha }~{\leftarrow }$ Learning rate, ${G_{D}}~{\leftarrow }$ Gradient descent optimizer, ${V_{s}}~{\leftarrow }$ Validation set, ${S_{arc}^{t}}~{\leftarrow }$ Network architecture parameters

3:

Output: Performance Parameter,${O_{pp}}{\leftarrow }$ [RMSE,Score]

4:

for $i = (1 \hspace {1mm} to\hspace {1mm} E)$ do

5:

for $j = (1 \hspace {1mm} to\hspace {1mm} N)$ do

6:

${\delta _{C}}$ = Correlation Analysis (${x_{j}^{i}}$ ,${y_{j}^{i}}$ )

end ${for^{j}}$

end ${for^{i}}$

7:

for $k = (1 \hspace {1mm} to\hspace {1mm} M)$ do

8:

${\delta _{N}}$ = Median Filter(${\delta _{C}^{k}}$ ,W)

9:

${\delta _{D}}$ = Data Normalization(${\delta _{N}^{k}}$ , mean(${\delta _{N}}$ ), SD(${\delta _{N}}$ ))

10:

${\delta _{P}}$ = Piece-wise linear function(${\delta _{C}^{k}}$ )

end ${for^{k}}$

11:

${S_{arc}^{t}}~{\leftarrow }$ (${L_{N}}$ ,${L_{H}}$ ,${L_{F}}$ ,${\alpha }$ ,${G_{D}}$ ,${L_{D}}$ ,${V_{s}}$ ,${R_{L}}$ ,${\delta _{D}}$ ,${\delta _{P}}$ )

12:

for $l = (1 \hspace {1mm} to\hspace {1mm} T)$ do

13:

${F_{D}}$ =Forward Pass (${S_{arc}^{t}}$ )

14:

${E_{t}}$ = Error(${F_{D}}$ )

15:

${B_{p}}$ = Back Propagation(${E_{t}}$ )

16:

${W_{new}}$ = ${W_{old}}$ + ${\alpha }$ *derivative(${B_{p}}$ )

17:

${b_{new}}$ = ${b_{old}}$ + ${\alpha }$ *derivative(${B_{p}}$ )

end ${for^{l}}$

18:

${O_{pp}^{i}}{\leftarrow }$ Calculate RMSE using equation 14.

19:

${O_{pp}^{i+1}}{\leftarrow }$ Calculate score function using equation 15.

I. Hyper Parameters Selection

LSTM training process involves many different parameters. These parameters have a great impact on the accuracy of the model. These parameters include learning rate, batch size, number of layers, number of neurons in each layer, optimizer and the training epochs. There is no definite rule or process by which the best values of these parameters are selected, generally an iterative grid search method is used to get the best results. The model is run against different values of these parameters and combination of values with best result are recorded. Addition of more layers makes system more complex and timing consuming for training but could result in better predictions as compared to the single layer network which are more suited to the simple problems. The ranges/types of hyper parameters used for LSTM network training are given in table 6.

TABLE 6 Ranges/Types of Hyper Parameters in LSTM Model Training

J. Evaluation Metrics

After network development, training of LSTM network is carried out and selection of hyper parameters is made on the basis of root mean square error (RMSE) between actual RUL and predicted RUL for each engine of the turbofan dataset. Our proposed LSTM model is evaluated using two widely used evaluation methods: RMSE and scoring function. RMSE is the most used evaluation metric for predictions used by the researcher for past decades. It is a symmetrical scoring function which means it can assign equals weights or penalties to both early and late prediction. The equation of RMSE is given below.

$$\begin{equation*} RMSE=\sqrt {\frac {\sum _{i=1}^{N} e_{i}^{2}}{N}}\tag{14}\end{equation*}$$ where $\mathit {e_{i}}$ , is the prediction error and ${N}$ is the total number of samples. In PHM08 data challenge competition, a score function was employed to evaluate the performance of the prediction model [61]. It is an asymmetrical score function which means it can assign more weights to late prediction as opposed to early prediction. It is described mathematically as: $$\begin{align*} score=&{\sum _{j=1}^{N} s_{j}} \tag{15}\\ {s_{j}}=&\begin{cases} {e^{\frac {-h_{j}} {13} }} - 1, & {\text{if }{h_{j}} < 0} \\ {e^{\frac {-h_{j}} {10} }} - 1, & {\text{if }{h_{j}}\geq 0} \end{cases}\tag{16}\end{align*}$$ The performance of the model will degrade at greater extent in case of delayed prediction as compared to the early prediction. The prediction error between the actual and predicted RUL is close to zero then the value of score and RMSE is also smaller with advancing towards zero.

SECTION V. Results and Discussion

In this section, a detail analysis on RUL prediction results by our proposed model is presented. There are various parameters that affects the model’s accuracy. We can categorized them into two sets, first are the parameters related to the initial RUL value, these includes window size and threshold values. Second ones are LSTM hyper parameters. Combining these two sets of parameters gives a very large set of parameters affecting the model’s accuracy. It is evident from the existing literature on deep learning techniques that there is no straight forward way of determining a single best set of values for these parameters. This requires very extensive and detailed simulations to analyze it further and an iterative grid search approach is followed. The simulation model is developed in MATLAB and hundreds of runs were carried out against the parameters ranges defined in the last section. For comparison, four LSTM models which gave us encouraging results were further tested and tuned for best performance. The models along with the hyper parameters are given in table-7, LSTM model structure remains same as described in the Fig. 1. The output of the pre-processing stages which processed sensor data and RUL output labels are given as an input into the LSTM network. The network is trained using the different values of the hyper parameters given in the table-7 and the window size and threshold ranges defined in the last section. After following an iterative grid search approach, the best hyper parameters are selected for the proposed model on the basis of lowest RMSE and score function values. The results of the these four LSTM models against different thresholds and window sizes values are given in table 8.

TABLE 7 Selected LSTM Models With Hyper-Parameters Values

TABLE 8 Model Accuracy in Term of RMSE and Score Function Values for the Four LSTM Models

The results in table-8 show that there is no single set of best values parameters for the whole dataset. The best values of parameters for RMSE and score function are different for each sub-unit. For FD001 case the best results are with using LSTM Model-2 and window size and threshold values of 12 and 0.2 respectively. Similarly, for FD002 best results are with using LSTM Model-2 and window size and threshold values of 12 and 0.2 respectively. For FD003, best results are with LSTM Model-3 and window size and threshold values of 12 and 0.2 respectively and for FD004, best results are with using LSTM Model-1 and window size and threshold values of 12 and 0.2 respectively. It can be observed that although LSTM model hyper parameters are different for different sub-units, the best values of window size and threshold are same. The minimum initial RUL value among all engines in a sub dataset is selected as its initial RUL. The initial RUL values achieved from the training part of sub-units FD001 to FD004 are 78, 103, 79 and 87 respectively. These values are then used in the testing stage for each sub-unit. So, instead of using a single initial RUL value for the entire dataset which is used in most of the existing literature, we can achieve improve prediction accuracy by using separate initial RUL value and LSTM hyper parameters values for each sub-unit.

The prediction results using the best models are shown in Fig. 5, which shows highly accurate RUL predictions. The ${x}$ -axis denotes the number of engines in a specific sub-unit and ${y}$ -axis is the result of RUL prediction for each engine. The red and blue legends on the graph indicate the predicted and true value of RUL. Moreover, these results are further observed by histogram distribution of prediction error in Fig. 6.

FIGURE 5.

RUL Prediction on Test Set, (a) FD001 (b) FD002, (c) FD003, (d) FD004.

FIGURE 6.

Histogram Distribution of Error, (a) FD001, (b) FD002, (c) FD003, (d) FD004.

The histogram distribution of four sub-units indicates the variation of RMSE across the dataset. The ${x}$ -axis indicates the error or difference between predicted observation and true RUL while the ${y}$ -axis indicates the frequency of occurrence for the given error. It can be seen from the figure that large concentration for frequency of error lies in the range of [-5 0] in FD001 & FD003 while for FD002 and FD004, it is concentrated in [-10 0]. The data description in section III shows that FD002 and FD004 has 6 operating conditions with more than 200 tracking trajectories. In correlation analysis, we have demonstrated that these two sub-units possess irregular behavior so RUL prediction for these complex sequences is a challenge for the prediction model. FD001 has lowest RMSE and score because of one fault mode. Finally, we have compared our proposed LSTM model with other method published in the last few years on C-MAPSS sensor degradation dataset. The validation of model is investigated by RMSE and score function values which is presented in table 9. Our proposed model achieves state of the art performance on FD001 and FD003 sub-dataset with minimum RMSE, score values and it shows second best result on FD002 and FD004 sub-dataset. The minimum values of FD001 and FD003 are lower than FD002 and FD004 because of the irregular behavior of sensor degradation data. The accuracy significantly depends on the improved automatic piece wise linear degradation function which decides how early you need to provide the maintenance operation by determining the starting point of the degradation.

TABLE 9 Comparison of RMSE and Score With Other Methods

The robustness of our proposed model is showed by predicting the full life cycle time of few randomly selected test engines in Figures 7–​10. Fig. 7 shows the actual and predicted degradation results of RUL of four different types of engines from a total set of 100 engines for validating the performance of model for FD001. In a similar manner, Fig. 8–​10 shows the actual and predicted results of full cycle for other three sub-units. From these prediction graphs, we analyze that the predicted and actual degradation of test engine in FD001 and FD003 sub-dataset are very efficient and accurate as shown in Fig. 7 and Fig. 9 but for FD002 and FD004, we have seen that there was some irregular and abrupt changes in the predicted results while tracking the actual target RUL of test engine due to the complex nature of theses sub-datasets, this is shown in Fig. 8 and Fig. 10. In case of prediction on actual RUL having a plot similar to a straight line as shown in Fig. 9(b), our model performs quite good on tracking the degradation trends obtained by applying piecewise linear function on a target labels.

FIGURE 7.

RUL Prediction of 4 engines in FD001 sub-dataset (a) engine # 6 (b) engine # 20 (c) engine # 42 (d) engine # 90.

FIGURE 8.

RUL Prediction of 4 engines in FD002 sub-dataset (a) engine # 1 (b) engine # 40 (c) engine # 84 (d) engine # 150.

FIGURE 9.

RUL Prediction of 4 engines in FD003 sub-dataset (a) engine # 1 (b) engine # 18 (c) engine # 48 (d) engine # 75.

FIGURE 10.

RUL Prediction of 4 engines in FD004 sub-dataset (a) engine # 15 (b) engine # 73 (c) engine # 155 (d) engine # 202.

To conclude, in what follows we reason why the proposed prediction framework is able to yield better results than the existing equivalent models. The proposed framework consists of multiple stages from correlation analysis to LSTM network. The significant stages in this framework that greatly enhance the overall accuracy of model are correlation function to filter out irrelevant sensor variables as given in Fig. 2, and the estimation of initial RUL value with piecewise linear function as given in algorithm 1. This function gives the starting point of degradation for sensor data, and hence, we have used those values for training our deep LSTM network. Moreover, the hyper-parameters selection for the neural network plays a significant role on the effectiveness of model. Therefore, in our research work, the hyper-parameters have been picked through an iterative grid search based approach as shown in Table-6. Thus, these steps when combined in a single prediction pipeline achieved optimum results compared to the existing methods on all the four sub-units, and its tracking performance on the complete life cycle time prediction will be beneficial in preventing the real life failures of machines.

SECTION Algorithm 1 Improved Piece-Wise Linear Function for Initial RUL Calcultion for Each Engine

1:

Inputs: Time cycle values (tc) and Sensors data after filtering and normalization

2:

Parameters: ${w}$ = window length, ${g}$ = total number of windows, ${w_{1}}$ , ${w_{2}}$ … ${w_{g}}$ = subsequent windows, Th =threshold

3:

Output: Initial RUL label (irul)

4:

${e}$ = Extract sensors values from the given windows (${w_{1}}$ , ${w_{2}}$ … ${w_{g}}$ )

5:

${m}$ = Calculate the centroid of each window by computing the mean for each window

6:

for $i = (2 \hspace {1mm} to\hspace {1mm} g-1)$ do

7:

${s}$ = Subtract the mean of two windows (${w_{1}}$ , ${w_{i}}$ )

8:

sq = square (${s}$ )

9:

if sq >= Th then

10:

irul = tc - w * i

11:

else

12:

i = i+ 1

End

SECTION VI. Conclusion & Future Direction

This work presented a LSTM based model for predicting the RUL of turbofan engines. The proposed work shows that LSTM model when combined with effective pre-processing steps results in highly accurate RUL predictions. In addition to widely used pre-processing steps like feature selection, filtering and normalization, this work proposed an improved piecewise linear degradation model which estimated the initial RUL valuewhich is the starting point of the degradation. This value had a great impact on the RUL prediction accuracy. LSTM network of the proposed model which consisted of a combination of multiple LSTM layers, dropout layers, fully connected layers and a regression layer, was tested on multiple hyper parameter combinations to achieve the best result. The proposed model was tested on the C-MAPSS turbofan engine simulation dataset. the prediction accuracy depends upon various number of parameters which include parameters for initial RUL calculations and hyper parameters for LSTM model. It was observed that due to different dynamics of each sub dataset, it is better to use a separate prediction model for each sub-units instead of a single model for the whole dataset. RUL predictions of the proposed model were compared with recent benchmark publications and showed higher prediction accuracy. The threshold value and the window size are two important parameters in the degradation model which had great impact on the model accuracy. In future, this approach can be considered for online problems and other machines.

In future, we will further integrate novel deep learning algorithms into our prediction pipeline for processing the complex and real-world sensor data as opposed to traditional techniques. In the field of prognostics, normally we don’t get large amount of real-world data for industrial machines specially in-case of faulty conditions. Therfore, we need to develop such novel methods that can be able to train on small dataset, but, it predict the output effectively on large scale practical data. Basically, we requires such types of models, which have strong generalization capability with respect to output. Morever, for further improvements, we have to evaluate the performance of models on other publicly available dataset.