Robust Spatiotemporal Fusion of Satellite Images: A Constrained Convex Optimization Approach

This article proposes a novel spatiotemporal (ST) fusion framework for satellite images, named robust optimization-based ST fusion (ROSTF). ST fusion is a promising approach to resolve a tradeoff between the temporal and spatial resolution of satellite images. Although many ST fusion methods have been proposed, most of them are not designed to explicitly account for noise in observed images, despite the inevitable influence of noise caused by the measurement equipment and environment. Our ROSTF addresses this challenge by formulating noise removal and ST fusion as a unified optimization problem. First, we define observation models for satellite images that may be contaminated with random noise, outliers, and/or missing values. Next, we introduce certain assumptions that naturally hold between the observed images and the target high-resolution image. Then, based on these models and assumptions, we formulate the fusion problem as a constrained optimization problem and develop an efficient algorithm based on a preconditioned primal–dual splitting method (P-PDS) for solving the problem. The performance of ROSTF was verified using simulated and real data. The results show that ROSTF performs comparably to several state-of-the-art ST fusion methods in noiseless cases and outperforms them in noisy cases.

SECTION I. Introduction

The analysis of temporal image series is necessary and important in many remote sensing applications, such as vegetation/crop monitoring and estimation [1], evapotranspiration estimation [2], atmosphere monitoring [3], land-cover/land-use change detection [4], surface dynamic mapping [5], ecosystem monitoring [6], soil water content analysis [7], and detailed analysis of human–nature interactions [8]. These applications require time series of high-spatial-resolution (HR) images to properly model the ground surface. In addition, time series of high-temporal-resolution images are also needed to capture the changes in the ground surface that occur over short periods of time.

However, there is a tradeoff between the temporal and spatial resolution of satellite sensors, and no single sensor can satisfy both requirements simultaneously. For example, the Landsat sensors can acquire images with an HR of 30 m, but they have a revisit period of up to 16 dates. On the other hand, the Moderate Resolution Imaging Spectroradiometer (MODIS) sensors can acquire images for the same scene at least once per date, but the images are at a low spatial resolution (LR) of 500 m [9]. Therefore, the simultaneous acquisition of image series of high spatial and high temporal resolution is a major challenge in the remote sensing community [10]. A simple solution to this challenge is to perform super-resolution on the corresponding single LR image to estimate the unobserved HR image [11], [12]. However, it is too difficult because the spatial resolution gap between the two satellite images is often quite large.

Spatiotemporal (ST) fusion (ST fusion) addresses this challenge by utilizing pairs of HR and LR images taken on reference dates that are temporally close to the target date. Specifically, the unobserved HR image on the target date is estimated by combining detailed spatial structure extracted from the HR images on the reference dates and spectral changes captured from the differences between the LR images on the reference and target dates. In ideal situations, where a large number of reliable reference images are available, achieving accurate ST fusion would be straightforward because the correct spatial structure and spectral changes are readily available. In real-world applications, however, such situations are very rare. Therefore, to achieve the desired ST fusion in real-world applications, the following two requirements are important.

1. Minimum Reference Dates: ST fusion methods that require minimum reference dates are preferred. In many remote sensing applications, only one pair of images on a reference date may be available due to cloud contamination, inconsistencies in image acquisition timing, or other factors. In addition, preparing another pair of images can be time-consuming. Therefore, ST fusion methods using a single pair of HR and LR images on a reference date apply to a wider range of cases than those using multiple pairs, although such a situation is obviously challenging [13].

2. Robustness to Noise: Due to the measurement equipment and/or environment, satellite images are often contaminated with various types of noise, such as random noise, outliers, and missing values [14], [15]. An estimated HR image that remains noisy would significantly affect subsequent processing. Therefore, it is imperative to develop noise-robust ST fusion methods.

A. Prior Research

Many ST fusion methods have been proposed over the past decades. They are generally categorized into five groups [16]: unmixing-based, weight-function-based, Bayesian-based, learning-based, and hybrid methods. Unmixing-based methods estimate the pixel values of an HR image by unmixing the pixels of the input LR images based on the linear spectral mixing theory [17], [18]. Weight-function-based methods generate an HR image by combining information of all the input images based on weight functions [9], [19]. Bayesian-based methods use Bayesian estimation theory to fuse the input images in a probabilistic manner [20], [21]. In the Bayesian framework, ST fusion can be considered a maximum a posteriori (MAP) problem, that is, the goal is to obtain an HR image on the target date by maximizing its conditional probability relative to the input HR and LR images. Learning-based methods model the relationship between observed HR and LR image pairs and then predict an unobserved HR image using machine-learning algorithms such as sparse representation learning [22], [23], random forest [24], and neural networks [25], [26], [27], [28], [29], [30], [31]. Hybrid methods integrate two or more techniques from the above four categories [32], [33].

Some of the unmixing-based, weight-function-based, and Bayesian-based methods allow an arbitrary number of reference dates. In other words, they can handle the cases with a single reference date. However, they are sensitive to noise because they estimate an HR image at the pixel level based entirely on reference images, which may be noisy. Thus, if the input images are noisy, the output image will be severely degraded.

On the other hand, in the context of learning-based methods, a robust ST fusion network (RSFN) [28] has been established to account for Gaussian noise. RSFN automatically filters noise and prevents predictions from being corrupted by using convolutional neural networks (CNNs), generative adversarial networks (GANs), and an attention mechanism. Specifically, the RSFN uses the attention mechanism to ignore noisy pixels in two reference HR images and focus on clean pixels. This method only works in situations where noisy pixels in the two reference HR images do not appear at the same location. In real-world measurements, however, such situations are rare because noise generally contaminates the entire image, not just parts of it. Furthermore, as mentioned above, RSFN requires two reference dates.

B. Contributions and Paper Organization

Now, a natural question arises: How to achieve robust ST fusion with only a single reference date? In this article, we propose a novel ST fusion framework for satellite images, named robust optimization-based ST fusion (ROSTF), to estimate an HR image on the target date while simultaneously denoising an HR image on a single reference date.

Before formulating our optimization problem, we newly define two observation models (they will be detailed in Section III-A).

1. The first model describes the relationship between an observed noisy image and the oracle noiseless image. The model is designed under the assumption that the observed image is not only contaminated with random noise but also with outliers and missing values. Specifically, random noise is modeled by Gaussian noise while outliers and missing values are modeled by sparse noise.

2. The second model represents the relationship between an HR image and the corresponding LR image, based on a super-resolution model [34].

We also introduce the following two assumptions about satellite images (they will be detailed in Section III-B).

1. The reflectance may change significantly between the reference and target dates, but the land structure (the locations of the edges) does not. This is a very natural assumption in the context of ST fusion.

2. An HR image and the corresponding LR image have similar brightness. This assumption is necessarily true if the HR and LR sensors have similar spectral resolutions, as is the case with sensors like Landsat and MODIS [9].

Based on the observation models and assumptions, we formulate the fusion problem as a constrained convex optimization problem. Subsequently, we develop an efficient algorithm based on the preconditioned primal–dual splitting method (P-PDS) [35] with an operator-norm-based design method of variable-wise diagonal preconditioning, named OVDP [36], which can automatically determine the appropriate stepsizes for solving the optimization problem.

The main contributions of the article are given as follows.

1. Robustness to Random Noise, Outliers, and Missing Values: As described above, no existing ST fusion methods can handle random noise, outliers, and missing values, although this type of noise contaminates satellite images due to the measurement equipment and environment. Thanks to the formulation that incorporates the first observation model developed in Section III-A, ROSTF is robust to such mixed noise.

2. Single Reference Date: Assumption 1) is very simple but effective for ST fusion. By incorporating this assumption as a constraint in the optimization problem, we realize the mechanism to promote the estimated HR image on the target date and the denoised HR image on the reference date to have edges at similar locations. Thanks to such a mechanism, ROSTF performs well even based on a single reference date.

3. Facilitation of Parameter Adjustment: The objective function of the optimization problem of ROSTF consists only of image regularization terms to promote spatial piecewise smoothness. The other components, corresponding to data fidelity based on the two observation models and our two assumptions, are imposed as hard constraints. Such a formulation using constraints instead of adding terms to the objective function has the advantage of simplifying parameter setting [37], [38], [39], [40], [41], [42]: the appropriate parameters in the constraints do not depend on each other and can be determined independently for each constraint.

4. Automatic Stepsize Adjustment: To solve our optimization problem for ST fusion, we develop an efficient algorithm based on P-PDS with OVDP. The appropriate stepsizes of the standard PDS [43] (and most other optimization methods) would be different depending on the problem structure, which means that we have to adjust them manually. On the other hand, P-PDS with OVDP can automatically determine the appropriate stepsizes based on the problem structure, and thus our algorithm is free from such a troublesome task.

This paper is organized as follows. We first cover the mathematical preliminaries of our method in Section II and then proceed to the establishment of our method in Section III. In Section IV, we demonstrate the performance of ROSTF and the effectiveness of each key component of ROSTF through comparative experiments and ablation studies, respectively. Experimental results show that ROSTF performs comparably to several state-of-the-art ST fusion methods for noiseless images and outperforms them for noisy images, thanks to the effective work of each key component.

The preliminary version of this work, without considering sparse noise, mathematical details, comprehensive experimental comparison, deeper discussion, or implementation using P-PDS with OVDP, has appeared in conference proceedings [44].

SECTION II. Preliminaries

A. Notations

Let $\mathbb {R}$ be the set of all real numbers. Vectors and matrices are denoted by bold lower and upper case letters, respectively. We treat a multispectral image with spatial resolution $N_{1} \times N_{2}$ and spectral resolution (the number of bands) B as a vector $\mathbf {x} = ([\mathbf {x}]_{1}^{\top },\ldots,[\mathbf {x}]_{B}^{\top })^{\top }\in \mathbb {R}^{NB} (N=N_{1} N_{2})$ , where $[\mathbf {x}]_{b} \in \mathbb {R}^{N}$ is the vector representing the bth band pixel values. Here, the nth pixel of $[\mathbf {x}]_{b}$ is denoted by $x_{b,n} \in \mathbb {R}$ . Let $\Gamma _{0}(\mathbb {R}^{NB})$ be the set of all proper lower-semicontinuous convex functions defined on $\mathbb {R}^{NB}$ . The $\ell _{1}$ -norm, $\ell _{2}$ -norm, and $\ell _{1,2}$ -norm of $\mathbf {x}$ are defined as $\| {\mathbf x} \|_{1}:=\sum _{b}\sum _{n}|x_{b,n}|$ , $\| {\mathbf x} \|_{2}:= \left({\sum _{b}\sum _{n}|x_{b,n}|^{2}}\right)^{1/2}$ , and $\| {\mathbf x} \|_{1,2}:=\sum _{n} \left({\sum _{b}|x_{b,n}|^{2}}\right)^{1/2}$ , respectively. For an image ${\mathbf x}\in \mathbb {R}^{NB}$ , let $\mathbf {D}_{v}$ and $\mathbf {D}_{h}\in \mathbb {R}^{NB \times NB}$ denote the matrices for computing the differences between vertical/horizontal adjacent pixels, respectively, and let $\mathbf {D}:= (\mathbf {D}_{v}^{\top } \mathbf {D}_{h}^{\top })^{\top } \in \mathbb {R}^{2NB \times NB}$ . The hyperslab centered at $\omega $ with a radius $\alpha $ is denoted as \begin{equation*} S_{\alpha }^{\omega }:= \left \{{ \mathbf {z} | \,| \omega - \mathbf {1}^{\top }\mathbf {z} | \leq \alpha }\right \}. \tag{1}\end{equation*} Also, the $\ell _{p}$ -norm ball ($p=1, 2$ ) with the center $\mathbf {c}$ and radius $\varepsilon $ is denoted as \begin{equation*} B_{p, \varepsilon }^{\mathbf {c}}:= \left \{{ \mathbf {z} | \| \mathbf {z} - \mathbf {c} \|_{p} \leq \varepsilon }\right \}. \tag{2}\end{equation*} The indicator function $\iota _{C}: \mathbb {R}^{N} \rightarrow (-\infty,\infty]$ of a nonempty closed convex set C is defined as \begin{align*} \iota _{C}:= \begin{cases} \displaystyle 0, & \mathrm {if} \, \mathbf {x} \in C, \\ \displaystyle \infty, & \mathrm {otherwise}. \end{cases} \tag{3}\end{align*}

B. Proximal Tools

The optimization problem of ROSTF, to be formulated in Section III-B, involves nonsmooth convex functions. To solve such a problem, we introduce the notion of the proximity operator of index $\gamma > 0$ of $f \in \Gamma _{0}(\mathbb {R}^{NB})$ as follows:\begin{align*} \mathrm {prox}_{\gamma f}: \mathbb {R}^{NB} \rightarrow \mathbb {R}^{NB}: \mathbf {x} \mapsto \mathop {\mathrm{ argmin}}\limits _{\mathbf {y} \in \mathbb {R}^{NB}} f\left ({\mathbf {y}}\right) + \frac {1}{2\gamma }\|\mathbf {x} - \mathbf {y}\|_{2}^{2}. \tag{4}\end{align*}

The Fenchell–Rockerfellar conjugate function $f^{\ast}$ of the function $f \in \Gamma _{0}(\mathbb {R}^{NB})$ is denoted by \begin{equation*} f^{\ast}\left ({\mathbf {y}}\right):= \sup _{\mathbf {x}\in \mathbb {R}^{NB}} \left \{{ \langle \mathbf {x},\mathbf {y}\rangle - f\left ({\mathbf {x}}\right) }\right \}. \tag{5}\end{equation*} Thanks to Moreau’s identity [45], the proximity operator of $f^{\ast}$ is efficiently calculated as \begin{equation*} \mathrm {prox}_{\gamma f^{\ast}}\left ({{\mathbf x}}\right) = {\mathbf x}- \gamma \mathrm {prox} _{\frac {1}{\gamma }f}\left ({\tfrac {1}{\gamma } {\mathbf x}}\right). \tag{6}\end{equation*}

Below, we present the specific proximity operators of the functions used in this article. The proximity operator of the $\ell _{1,2}$ -norm is given by \begin{equation*} \left [{\mathrm {prox}_{\gamma \|\cdot \|_{1,2}}\left ({\mathbf {x}}\right)}\right]_{b,n} = \max \left \{{ 1 - \frac {\gamma }{\sqrt {\sum _{b'=1}^{B} |x_{b',n}|^{2}}},0 }\right \} x_{b,n}. \tag{7}\end{equation*} The proximity operator of the indicator function of the hyperslab in (1) is expressed as follows:\begin{align*} \mathrm {prox}_{\gamma \iota _{S_{\alpha }^{\omega }}}\left ({\mathbf {x}}\right) = \begin{cases} \displaystyle \mathbf {x} + \frac {\eta _{1} - \mathbf {1}^{\top }\mathbf {x}}{NB}\mathbf {1}, \; \; & \mathrm {if} \, \mathbf {1}^{\top }\mathbf {x} < \eta _{1}, \\ \displaystyle \mathbf {x} + \frac {\eta _{2} - \mathbf {1}^{\top }\mathbf {x}}{NB}\mathbf {1}, \; \; & \mathrm {if} \, \mathbf {1}^{\top }\mathbf {x} > \eta _{2}, \\ \displaystyle \mathbf {x}, & \mathrm {otherwise} \end{cases} \tag{8}\end{align*} where $\eta _{1} = \omega - \alpha $ and $\eta _{2} = \omega + \alpha $ . The proximity operators of the indicator functions of the $\ell _{2}$ -norm ball and the $\ell _{1}$ -norm ball in (2) are calculated by \begin{align*} \mathrm {prox}_{\gamma \iota _{B_{2, \varepsilon }^{\mathbf {c}}}}\left ({\mathbf {x}}\right) = \begin{cases} \displaystyle \mathbf {x}, & \mathrm {if} \, \mathbf {x} \in \iota _{B_{2, \varepsilon }^{\mathbf {c}}}, \\ \displaystyle \mathbf {c} + \frac {\varepsilon \left ({\mathbf {x} - \mathbf {c}}\right)}{\| \mathbf {x} - \mathbf {c} \|_{2}}, & \mathrm {otherwise} \end{cases} \tag{9}\end{align*} and a fast $\ell _{1}$ -ball projection algorithm [46], respectively.

C. P-PDS With OVDP

The standard PDS [43] is a versatile and efficient proximal algorithm that can solve a wide class of nonsmooth convex optimization problems without using matrix inversion. However, it is troublesome to manually set the appropriate stepsizes of the standard PDS. Therefore, we adopt P-PDS [35] with OVDP [36], a method that automatically determines the appropriate stepsizes according to the problem structure.

Let ${\mathbf y}_{i} \in \mathbb {R}^{K_{i}} (i = 1, {\dots }, N)$ and ${\mathbf z}_{j} \in \mathbb {R}^{L_{j}} (j = 1, {\dots }, M)$ . Consider convex optimization problems of the following form:\begin{align*} \min _{\substack {\mathbf y_{1}, {\dots }, {\mathbf y}_{N} \\ {\mathbf z}_{1}, {\dots }, {\mathbf z}_{M}}} & \sum _{i=1}^{N}g_{i}\left ({{\mathbf y}_{i}}\right)+\sum _{j=1}^{M}h_{j}\left ({{\mathbf z}_{j}}\right), \\ {\mathrm{ s.t.}} & \begin{cases} \displaystyle {\mathbf z} _{1} = \sum \nolimits _{i=1}^{N} \mathbf {G}_{1,i} {\mathbf y}_{i}, \\ \displaystyle \vdots \\ \displaystyle {\mathbf z} _{M} = \sum \nolimits _{i=1}^{N} \mathbf {G}_{M,i} {\mathbf y}_{i} \end{cases} \tag{10}\end{align*} where $g_{i}\in \Gamma _{0}(\mathbb {R}^{K_{i}}) (i = 1, {\dots }, N), h_{j}\in \Gamma _{0}(\mathbb {R}^{L_{j}}) (j = 1, {\dots }, M)$ , and $\mathbf {G}_{j,i}: \mathbb {R}^{K_{i}} \rightarrow \mathbb {R}^{L_{j}} (i=1, {\dots },N, j=1, {\dots },M)$ are linear operators. P-PDS with OVDP solves (10) by the following iterative procedures:\begin{align*} \begin{cases} \displaystyle \bar {\mathbf y}_{1}^{\left ({n}\right)} &\leftarrow {\mathbf y} _{1}^{\left ({n}\right)}-\gamma _{1,1}\sum \nolimits _{j=1}^{M}\mathbf {G}_{j,1}^{\top } {\mathbf z}_{j}^{\left ({n}\right)}, \\ \displaystyle {\mathbf y} _{1}^{\left ({n+1}\right)} &\leftarrow \mathrm {prox} _{\gamma _{1,1}g_{1}}\left ({\bar {\mathbf y}_{1}^{\left ({n}\right)}}\right), \\ \displaystyle &\vdots \\ \displaystyle \bar {\mathbf y}_{N}^{\left ({n}\right)} &\leftarrow {\mathbf y} _{N}^{\left ({n}\right)}-\gamma _{1,N}\sum \nolimits _{j=1}^{M}\mathbf {G}_{j,N}^{\top } {\mathbf z}_{j}^{\left ({n}\right)}, \\ \displaystyle {\mathbf y} _{N}^{\left ({n+1}\right)} &\leftarrow \mathrm {prox} _{\gamma _{1,N}g_{N}}\left ({\bar {\mathbf y}_{N}^{\left ({n}\right)}}\right), \\ \displaystyle \bar {\mathbf z}_{1}^{\left ({n}\right)} &\leftarrow {\mathbf z} _{1}^{\left ({n}\right)} + \gamma _{2,1}\sum \nolimits _{i=1}^{N}\mathbf {G}_{1,i}\left ({2 {\mathbf y}_{i}^{\left ({n+1}\right)}- {\mathbf y}_{i}^{\left ({n}\right)}}\right), \\ \displaystyle {\mathbf z} _{1}^{\left ({n+1}\right)} &\leftarrow \mathrm {prox} _{\gamma _{2,1}h_{1}^{\ast}}\left ({\bar {\mathbf z}_{1}^{\left ({n}\right)}}\right), \\ \displaystyle &\vdots \\ \displaystyle \bar {\mathbf z}_{M}^{\left ({n}\right)} &\leftarrow {\mathbf z} _{M}^{\left ({n}\right)} + \gamma _{2,M}\sum \nolimits _{i=1}^{N}\mathbf {G}_{M,i}\left ({2 {\mathbf y}_{i}^{\left ({n+1}\right)}- {\mathbf y}_{i}^{\left ({n}\right)}}\right), \\ \displaystyle {\mathbf z} _{M}^{\left ({n+1}\right)} &\leftarrow \mathrm {prox} _{\gamma _{2,M}h_{M}^{\ast}}\left ({\bar {\mathbf z}_{M}^{\left ({n}\right)}}\right) \end{cases}\end{align*} where $\gamma _{1,i} (i=1, {\dots },N)$ and $\gamma _{2,j} (j=1, {\dots },M)$ are stepsize parameters. The stepsize parameters can be determined as follows [36]:\begin{equation*} \gamma _{1,i} = \frac {1}{\sum _{j=1}^{M} \left \|{ \mathbf {G}_{j,i}}\right \|_{\mathrm {op}}^{2}},\quad \gamma _{2,j} = \frac {1}{N} \tag{11}\end{equation*} where $\|\cdot \|_{\mathrm {op}}$ represents the operator norm defined by \begin{equation*} \left \|{\mathbf {G}}\right \|_{\mathrm {op}}:= \sup _{\mathbf x\neq \mathbf {0}}\frac {\left \|{\mathbf {G} {\mathbf x}}\right \|_{2}} {\left \|{ {\mathbf x}}\right \|_{2}}. \tag{12}\end{equation*}

SECTION III. Proposed Method

From now on, we will focus on cases with a single reference date. Specifically, we consider a situation where both HR and LR sensors observe the same scene on the single reference date, but on the target date, only the LR sensor observes that scene and not the HR sensor. Let the HR image on the reference date, the LR image on the reference date, and the LR image on the target date be $\mathbf {h}_{r}\in \mathbb {R}^{N_{h} B}$ , $\mathbf {l}_{r}\in \mathbb {R}^{N_{l} B}$ , and $\mathbf {l}_{t}\in \mathbb {R}^{N_{l} B}$ , respectively. Our method, ROSTF, estimates the desired noiseless HR image on the target date, denoted by $\widehat {\mathbf {h}}_{t}\in \mathbb {R}^{N_{h} B}$ , based on these three observed images, while simultaneously denoising $\mathbf {h}_{r}$ . A general diagram of ROSTF is shown in Fig. 1.

Fig. 1.

Illustration of our method, that is, ROSTF.

A. Observation Models

Let $\widehat {\mathbf {h}} \in \mathbb {R}^{N_{h} B}$ and $\widehat {\mathbf {l}} \in \mathbb {R}^{N_{l} B}$ be a noiseless HR image and a noiseless LR image, respectively, taken on the same date. We introduce observation models for a noisy HR image $\mathbf {h} \in \mathbb {R}^{N_{h} B}$ and a noisy LR image $\mathbf {l} \in \mathbb {R}^{N_{l} B}$ . Specifically, we consider that the observed satellite images $\mathbf {h}$ and $\mathbf {l}$ are possibly contaminated with random noise, outliers, and missing values. Random noise added to $\widehat {\mathbf {h}}$ and $\widehat {\mathbf {l}}$ is modeled by Gaussian noise $\mathbf {n}_{h}$ and $\mathbf {n}_{l}$ with standard deviation $\sigma _{h}$ and $\sigma _{l}$ , respectively, while outliers and missing values affecting $\widehat {\mathbf {h}}$ and $\widehat {\mathbf {l}}$ are modeled by sparsely distributed noise $\mathbf {s}_{h}$ and $\mathbf {s}_{l}$ with the superimposition ratio $r_{h}$ and $r_{l}$ , respectively. By modeling the noise in this manner, the observation models for $\mathbf {h}$ and $\mathbf {l}$ are described as \begin{align*} \mathbf {h} &= \widehat {\mathbf {h}} + \mathbf {n}_{h}+ \mathbf {s}_{h}, \\ \mathbf {l} &= \widehat {\mathbf {l}} + \mathbf {n}_{l}+ \mathbf {s}_{l}. \tag{13}\end{align*} Here, $\sigma _{h} > \sigma _{l}$ and $r_{h} > r_{l}$ generally hold since HR images often contain more severe noise than LR images. This is because the amount of light received per pixel decreases as the number of pixels increases [47].

On the other hand, $\widehat {\mathbf {l}}$ can be approximated by the image obtained by blurring and downsampling $\widehat {\mathbf {h}}$ , known as a typical super-resolution model [34], as follows:\begin{equation*} \widehat {\mathbf {l}} = \mathbf {S}\mathbf {B}\widehat {\mathbf {h}} + \mathbf {m} \tag{14}\end{equation*} where $\mathbf {B} \in \mathbb {R}^{N_{h} B \times N_{h} B}$ is the spatial spread transform matrix introduced in [48], $\mathbf {S}\in \mathbb {R}^{N_{l} B \times N_{h} B}$ is the downsampling matrix, and $\mathbf {m} \in \mathbb {R}^{N_{l} B}$ is the modeling error. This model has been widely used in the ST fusion literature [49].

B. Problem Formulation

We introduce the following two assumptions about the noiseless HR and LR images on the reference and target dates, that is, $\widehat {\mathbf {h}}_{r}$ , $\widehat {\mathbf {l}}_{r}$ , $\widehat {\mathbf {h}}_{t}$ , and $\widehat {\mathbf {l}}_{t}$ .

1. The reflectance may change significantly between the reference and target dates, but the land structure does not. This is a very natural assumption in ST fusion and is implicitly accepted in previous studies. If the land structure has not changed significantly, the edges of $\widehat {\mathbf {h}}_{r}$ and $\widehat {\mathbf {h}}_{t}$ appear at almost the same locations, implying that the difference between $\mathbf {D}\widehat {\mathbf {h}}_{r}$ and $\mathbf {D}\widehat {\mathbf {h}}_{t}$ tends to be small. We measure the similarity of these edges using the $\ell _{p}$ ($p=1$ or 2) norm as $\| \mathbf {D}\widehat {\mathbf {h}}_{r} - \mathbf {D}\widehat {\mathbf {h}}_{t} \|_{p}$ .

2. The HR and LR images taken on the same date have similar average brightness per band. For example, the difference in average brightness of the bth band of $\widehat {\mathbf {h}}_{t}$ and $\widehat {\mathbf {l}}_{t}$ , expressed as \begin{equation*} \left |{\frac {1}{N_{l}}\mathbf {1}^{\top }\left [{\widehat {\mathbf {l}}_{t}}\right]_{b}-\frac {1}{N_{h}}\mathbf {1}^{\top }\left [{\widehat {\mathbf {h}}_{t}}\right]_{b}}\right | \tag{15}\end{equation*} is expected to be small. This is necessarily true if the HR and LR sensors have similar spectral resolutions, as is the case for Landsat and MODIS [9].

Based on these assumptions and the observation models in (13) and (14), we formulate the fusion problem as the following constrained convex optimization problem:\begin{align*} \min _{\substack { \widetilde {\mathbf {h}}_{r}, \widetilde {\mathbf {h}}_{t}, \\[4pt] \widetilde {\mathbf {s}}_{hr}, \widetilde {\mathbf {s}}_{lr}, \widetilde {\mathbf {s}}_{lt}}} \: & \left \|{\mathbf {D} \widetilde {\mathbf {h}}_{r}}\right \|_{1,2} + \lambda \left \|{\mathbf {D} \widetilde {\mathbf {h}}_{t}}\right \|_{1,2} \\[4pt] \mathrm {s.t.} \: & \begin{cases} \displaystyle \| \mathbf {D} \widetilde {\mathbf {h}}_{r} - \mathbf {D} \widetilde {\mathbf {h}}_{t} \|_{p} \leq \alpha, \\[4pt] \displaystyle \left |{ c_{b}- \mathbf {1}^{\top }\left [{\widetilde {\mathbf {h}}_{t}}\right]_{b}/{N_{h}}}\right | \leq \beta _{b} \, \left ({b = 1, \ldots, B}\right), \\[4pt] \displaystyle \| \mathbf {h}_{r}- \left ({\widetilde {\mathbf {h}}_{r}+ \widetilde {\mathbf {s}}_{hr}}\right) \|_{2}\leq \varepsilon _{h}, \\[2pt] \displaystyle \| \mathbf {l}_{r}- \left ({{ \mathbf {S}} \mathbf {B} \widetilde {\mathbf {h}}_{r} + \widetilde {\mathbf {s}}_{lr}}\right) \|_{2}\leq \varepsilon _{l},\\[1pt] \displaystyle \| \mathbf {l}_{t}- \left ({{ \mathbf {S}} \mathbf {B} \widetilde {\mathbf {h}}_{t} + \widetilde {\mathbf {s}}_{lt}}\right) \|_{2}\leq \varepsilon _{l},\\[2pt] \displaystyle \| \widetilde {\mathbf {s}}_{hr} \|_{1} \leq \eta _{h}, \\[2pt] \displaystyle \| \widetilde {\mathbf {s}}_{lr} \|_{1} \leq \eta _{l}, \\[2pt] \displaystyle \| \widetilde {\mathbf {s}}_{lt} \|_{1} \leq \eta _{l} \end{cases} \tag{16}\end{align*} where $\lambda $ is a balancing parameter. The variables $\widetilde {\mathbf {h}}_{r}$ and $\widetilde {\mathbf {h}}_{t}$ correspond to the estimates of $\widehat {\mathbf {h}}_{r}$ and $\widehat {\mathbf {h}}_{t}$ , respectively, and $\widetilde {\mathbf {s}}_{hr}$ , $\widetilde {\mathbf {s}}_{lr}$ , and $\widetilde {\mathbf {s}}_{lt}$ correspond to the estimates of sparse noise superimposed on $\mathbf {h}_{r}$ , $\mathbf {l}_{r}$ and $\mathbf {l}_{t}$ , respectively. Each term in the objective function and each constraint have the following roles.

1. The two terms in the objective function promote spatial piecewise smoothness of $\widetilde {\mathbf {h}}_{r}$ and $\widetilde {\mathbf {h}}_{t}$ , with the hyperspectral total variation (HTV) [50] as regularization.

2. The first constraint encourages $\mathbf {D}\widetilde {\mathbf {h}}_{r}$ and $\mathbf {D}\widetilde {\mathbf {h}}_{t}$ to be similar based on Assumption 1). The parameter $\alpha $ controls the degree of similarity. Hereafter, the constraint is referred to as the edge constraint.

3. The second constraint is designed based on Assumption 2). Since $\mathbf {l}_{t}$ is contaminated by noise, we do not use the average brightness of $[\mathbf {l}_{t}]_{b}$ itself, that is, $\mathbf {1}^{\top }[\mathbf {l}_{t}]_{b}/N_{l}$ , but the parameter $c_{b}$ , which is determined based on $\mathbf {1}^{\top }[\mathbf {l}_{t}]_{b}/N_{l}$ and the noise intensity. The parameter $\beta _{b}$ controls the strength of this constraint. Hereafter, the constraint is referred to as the brightness constraint.

4. The third constraint serves as data-fidelity based on the observation model in (13). The parameter $\varepsilon _{h}$ depends on the Gaussian noise intensity on the HR image, that is, $\sigma _{h}$ .

5. The fourth and fifth constraints are to ensure that the solutions follow the observation model in (14). The parameter $\varepsilon _{l}$ depends on the Gaussian noise intensity on the LR images, that is, $\sigma _{l}$ .

6. The last three constraints characterize the sparse noise using the $\ell _{1}$ norms. The parameters $\eta _{h}$ and $\eta _{l}$ depend on the sparse noise intensity on the HR and the LR images, respectively, that is, $r_{h}$ and $r_{l}$ .

Using constraints instead of adding terms to the objective function in this way simplifies the parameter setting [37], [38], [39], [40], [41], [42]: we can determine the appropriate parameters for each constraint independently because they are decoupled. The detailed setting of these parameters is discussed in Section IV-C.

C. Optimization

For solving (16) by an algorithm based on P-PDS with OVDP, we need to transform (16) into (10). First, using the hyperslab, the $\ell _{p}$ -norm ball, and the indicator function (see (1), (2), and (3) for the definition, respectively), we reformulate our problem in (16) as follows:\begin{align*} \min _{\substack { \widetilde {\mathbf {h}}_{r}, \widetilde {\mathbf {h}}_{t}, \\ \widetilde {\mathbf {s}}_{hr}, \widetilde {\mathbf {s}}_{lr}, \widetilde {\mathbf {s}}_{lt}}} & \left \|{\mathbf {D} \widetilde {\mathbf {h}}_{r}}\right \|_{1,2} + \lambda \left \|{\mathbf {D} \widetilde {\mathbf {h}}_{t}}\right \|_{1,2} + \iota _{B_{p, \alpha }^{\mathbf {0}}}\left ({\mathbf {D} \widetilde {\mathbf {h}}_{r} - \mathbf {D} \widetilde {\mathbf {h}}_{t} }\right) \\ &\quad + \sum _{b=1}^{B}\iota _{S_{\beta _{b}'}^{c_{b}'}}\left ({\left [{ \widetilde {\mathbf {h}}_{t}}\right]_{b}}\right) + \iota _{B_{2,\varepsilon _{h}}^{ \mathbf {h}_{r}}}\left ({\widetilde {\mathbf {h}}_{r}+ \widetilde {\mathbf {s}}_{hr}}\right) \\ &\quad +\iota _{B_{2,\varepsilon _{l}}^{ \mathbf {l}_{r}}}\left ({{ \mathbf {S}} \mathbf {B} \widetilde {\mathbf {h}}_{r} + \widetilde {\mathbf {s}}_{lr}}\right) + \iota _{B_{2,\varepsilon _{l}}^{ \mathbf {l}_{t}}}\left ({{ \mathbf {S}} \mathbf {B} \widetilde {\mathbf {h}}_{t} + \widetilde {\mathbf {s}}_{lt}}\right) \\ &\quad + \iota _{B_{1, \eta _{h}}^{\mathbf {0}}}\left ({\widetilde {\mathbf {s}}_{hr}}\right) + \iota _{B_{1, \eta _{l}}^{\mathbf {0}}}\left ({\widetilde {\mathbf {s}}_{lr}}\right) + \iota _{B_{1, \eta _{l}}^{\mathbf {0}}}\left ({\widetilde {\mathbf {s}}_{lt}}\right) \tag{17}\end{align*} where $\beta _{b}'=\beta _{b} N_{h}$ and $c_{b}'=c_{b} N_{h}$ for $b = 1,\ldots,B$ . Introducing auxiliary variables ${\mathbf z}_{1}$ , ${\mathbf z}_{2}$ , ${\mathbf z}_{3}$ , ${\mathbf z}_{4}$ , ${\mathbf z}_{5}$ , and ${\mathbf z}_{6}$ , we can transform (17) into the following equivalent problem:\begin{align*} \min _{\substack { \widetilde {\mathbf {h}}_{r}, \widetilde {\mathbf {h}}_{t}, \\ \widetilde {\mathbf {s}}_{hr}, \widetilde {\mathbf {s}}_{lr}, \widetilde {\mathbf {s}}_{lt}}} & \left \|{ {\mathbf z}_{1}}\right \|_{1,2} + \lambda \left \|{ {\mathbf z}_{2}}\right \|_{1,2} + \iota _{B_{p, \alpha }^{\mathbf {0}}}\left ({{\mathbf z}_{3}}\right) + \sum _{b=1}^{B}\iota _{S_{\beta _{b}'}^{c_{b}'}}\left ({\left [{ \widetilde {\mathbf {h}}_{t}}\right]_{b}}\right) \\ &\quad + \iota _{B_{2,\varepsilon _{h}}^{ \mathbf {h}_{r}}}\left ({{\mathbf z}_{4}}\right) +\iota _{B_{2,\varepsilon _{l}}^{ \mathbf {l}_{r}}}\left ({{\mathbf z}_{5}}\right) + \iota _{B_{2,\varepsilon _{l}}^{ \mathbf {l}_{t}}}\left ({{\mathbf z}_{6}}\right) \\ &\quad + \iota _{B_{1, \eta _{h}}^{\mathbf {0}}}\left ({\widetilde {\mathbf {s}}_{hr}}\right) + \iota _{B_{1, \eta _{l}}^{\mathbf {0}}}\left ({\widetilde {\mathbf {s}}_{lr}}\right) + \iota _{B_{1, \eta _{l}}^{\mathbf {0}}}\left ({\widetilde {\mathbf {s}}_{lt}}\right) \\ \mathrm {s.t.} &\begin{cases} \displaystyle {\mathbf z} _{1} = \mathbf {D} \widetilde {\mathbf {h}}_{r}, \\ \displaystyle {\mathbf z} _{2} = \mathbf {D} \widetilde {\mathbf {h}}_{t}, \\ \displaystyle {\mathbf z} _{3} = \mathbf {D} \widetilde {\mathbf {h}}_{r} - \mathbf {D} \widetilde {\mathbf {h}}_{t}, \\ \displaystyle {\mathbf z} _{4} = \widetilde {\mathbf {h}}_{r}+ \widetilde {\mathbf {s}}_{hr}, \\ \displaystyle {\mathbf z} _{5} = { \mathbf {S}} \mathbf {B} \widetilde {\mathbf {h}}_{r} + \widetilde {\mathbf {s}}_{lr}, \\ \displaystyle {\mathbf z} _{6} = { \mathbf {S}} \mathbf {B} \widetilde {\mathbf {h}}_{t} + \widetilde {\mathbf {s}}_{lt}. \end{cases} \tag{18}\end{align*} Then, by defining \begin{align*} &g_{1}\left ({\widetilde {\mathbf {h}}_{r}}\right)=0, g_{2}\left ({\widetilde {\mathbf {h}}_{t}}\right)=\sum _{b=1}^{B}\iota _{S_{\beta _{b}'}^{c_{b}'}}\left ({\left [{\widetilde {\mathbf {h}}_{t}}\right]_{b}}\right), \\ &g_{3}\left ({\widetilde {\mathbf {s}}_{hr}}\right)=\iota _{B_{1, \eta _{h}}^{\mathbf {0}}}\left ({\widetilde {\mathbf {s}}_{hr}}\right), g_{4}\left ({\widetilde {\mathbf {s}}_{lr}}\right)=\iota _{B_{1, \eta _{l}}^{\mathbf {0}}}\left ({\widetilde {\mathbf {s}}_{lr}}\right), \\ &g_{5}\left ({\widetilde {\mathbf {s}}_{lt}}\right)=\iota _{B_{1, \eta _{l}}^{\mathbf {0}}}\left ({\widetilde {\mathbf {s}}_{lt}}\right), h_{1}\left ({{\mathbf z}_{1}}\right) = \left \|{ {\mathbf z}_{1}}\right \|_{1,2}, \\ &h_{2}\left ({{\mathbf z}_{2}}\right) = \lambda \| {\mathbf z}_{2}\|_{1,2}, h_{3}\left ({{\mathbf z}_{3}}\right) = \iota _{B_{p, \alpha }^{\mathbf {0}}}\left ({{\mathbf z}_{3}}\right), \\ &h_{4}\left ({{\mathbf z}_{4}}\right) = \iota _{B_{2,\varepsilon _{h}}^{ \mathbf {h}_{r}}}\left ({{\mathbf z}_{4}}\right), h_{5}\left ({{\mathbf z}_{5}}\right) = \iota _{B_{2,\varepsilon _{l}}^{ \mathbf {l}_{r}}}\left ({{\mathbf z}_{5}}\right), \\ &h_{6}\left ({{\mathbf z}_{6}}\right) = \iota _{B_{2,\varepsilon _{l}}^{ \mathbf {l}_{t}}}\left ({{\mathbf z}_{6}}\right) \tag{19}\end{align*} we reduce (10) to (18), that is, (16).

The algorithm for solving (16) is summarized in Algorithm 1. The stepsizes are determined based on OVDP [36] as follows:\begin{align*} \gamma _{1,1} &= \frac {1}{2\left \|{ \mathbf {D}}\right \|_{\mathrm {op}}^{2} + \left \|{ \mathbf {I}}\right \|_{\mathrm {op}}^{2} + \left \|{ { \mathbf {S}} \mathbf {B}}\right \|_{\mathrm {op}}^{2}} = \frac {1}{18}, \\ \gamma _{1,2} &= \frac {1}{2\left \|{ \mathbf {D}}\right \|_{\mathrm {op}}^{2} + \left \|{ { \mathbf {S}} \mathbf {B}}\right \|_{\mathrm {op}}^{2}} = \frac {1}{17}, \\ \gamma _{1,3} &=\gamma _{1,4} = \gamma _{1,5} = \frac {1}{\left \|{ \mathbf {I}}\right \|_{\mathrm {op}}^{2}} = 1, \\ \gamma _{2,i} &= \frac {1}{5}, for i = 1, {\dots }, 6. \tag{20}\end{align*}

Algorithm 1 P-PDS-Based Solver for (16)

Require: $\lambda, p, \alpha, \beta _{b}', c_{b}', \varepsilon _{h}, \varepsilon _{l}, \eta _{h}, \eta _{l}$

Ensure: $\widetilde {\mathbf {h}}_{r}^{(n)}, \widetilde {\mathbf {h}}_{t}^{(n)}, \widetilde {\mathbf {s}}_{hr}^{(n)}, \widetilde {\mathbf {s}}_{lr}^{(n)}, \widetilde {\mathbf {s}}_{lt}^{(n)}$

1:

Initialize $\widetilde {\mathbf {h}}_{r}^{(0)}, \widetilde {\mathbf {h}}_{t}^{(0)}, \widetilde {\mathbf {s}}_{hr}^{(0)}, \widetilde {\mathbf {s}}_{lr}^{(0)}, \widetilde {\mathbf {s}}_{lt}^{(0)}, \mathbf {z}_{j}^{(0)}(j=1, {\dots },6)$ ;

2:

Set $\gamma _{1,i} (i=1,\ldots, 3), \gamma _{2,j} (j=1,\ldots, 6)$ , as in (20);

3:

while until a stopping criterion is not satisfied do

4:

$\mathbf {u}_{r} \leftarrow \mathbf {D} ^{\top } {\mathbf z}_{1}^{(n)} + \mathbf {D}^{\top } {\mathbf z}_{3}^{(n)} + {\mathbf z}_{4}^{(n)} + \mathbf {B}^{\top } { \mathbf {S}}^{\top } {\mathbf z}_{5}^{(n)}$ ;

5:

$\mathbf {u}_{t} \leftarrow \mathbf {D} ^{\top } {\mathbf z}_{2}^{(n)} - \mathbf {D}^{\top } {\mathbf z}_{3}^{(n)} + \mathbf {B}^{\top } { \mathbf {S}}^{\top } {\mathbf z}_{6}^{(n)}$ ;

6:

$\widetilde {\mathbf {h}}_{r}^{(n+1)} \leftarrow \widetilde {\mathbf {h}}_{r} ^{(n)} - \gamma _{1,1} \mathbf {u}_{r}$ ;

7:

$\widetilde {\mathbf {h}}_{t}^{(n+1)}\leftarrow \widetilde {\mathbf {h}}_{t} ^{(n)}-\gamma _{1,2} \mathbf {u}_{t}$ ;

8:

for $b=1, {\dots }, B$ do

9:

$[\widetilde {\mathbf {h}}_{t}^{(n+1)}]_{b}\leftarrow \mathrm {prox} _{\iota _{S_{\beta _{b}'}^{c_{b}'}}}([\widetilde {\mathbf {h}}_{t}^{(n+1)}]_{b})$ ;

10:

end for

11:

$\widetilde {\mathbf {s}}_{hr}^{(n+1)} \leftarrow \mathrm {prox} _{\gamma _{1,3} \iota _{B_{1,\eta _{h}}^{0}}}(\widetilde {\mathbf {s}}_{hr}^{(n)} - \gamma _{1,3} {\mathbf z}_{4})$ ;

12:

$\widetilde {\mathbf {s}}_{lr}^{(n+1)} \leftarrow \mathrm {prox} _{\gamma _{1,4} \iota _{B_{1,\eta _{l}}^{0}}}(\widetilde {\mathbf {s}}_{lr}^{(n)} - \gamma _{1,4} {\mathbf z}_{5})$ ;

13:

$\widetilde {\mathbf {s}}_{lt}^{(n+1)} \leftarrow \mathrm {prox} _{\gamma _{1,5} \iota _{B_{1,\eta _{l}}^{0}}}(\widetilde {\mathbf {s}}_{lt}^{(n)} - \gamma _{1,5} {\mathbf z}_{6})$ ;

14:

$\mathbf {v}_{r} \leftarrow 2 \widetilde {\mathbf {h}}_{r}^{(n+1)} - \widetilde {\mathbf {h}}_{r}^{(n)}$ ;

15:

$\mathbf {v}_{t} \leftarrow 2 \widetilde {\mathbf {h}}_{t}^{(n+1)} - \widetilde {\mathbf {h}}_{t}^{(n)}$ ;

16:

$\mathbf {w}_{hr} \leftarrow 2 \widetilde {\mathbf {s}}_{hr}^{(n+1)} - \widetilde {\mathbf {s}}_{hr}^{(n)}$ ;

17:

$\mathbf {w}_{lr} \leftarrow 2 \widetilde {\mathbf {s}}_{lr}^{(n+1)} - \widetilde {\mathbf {s}}_{lr}^{(n)}$ ;

18:

$\mathbf {w}_{lt} \leftarrow 2 \widetilde {\mathbf {s}}_{lt}^{(n+1)} - \widetilde {\mathbf {s}}_{lt}^{(n)}$ ;

19:

${\mathbf z}_{1}^{(n)} \leftarrow {\mathbf z} _{1}^{(n)} + \gamma _{2,1} \mathbf {D} \mathbf {v} _{r}$ ;

20:

${\mathbf z}_{2}^{(n)} \leftarrow {\mathbf z} _{2}^{(n)} + \gamma _{2,2} \mathbf {D} \mathbf {v} _{t}$ ;

21:

${\mathbf z}_{3}^{(n)} \leftarrow {\mathbf z} _{3}^{(n)} + \gamma _{2,3}(\mathbf {D} \mathbf {v} _{r} - \mathbf {D} \mathbf {v} _{t})$ ;

22:

${\mathbf z}_{4}^{(n)} \leftarrow {\mathbf z} _{4}^{(n)} + \gamma _{2,4}(\mathbf {v}_{r} + \mathbf {w}_{hr})$ ;

23:

${\mathbf z}_{5}^{(n)} \leftarrow {\mathbf z} _{5}^{(n)} + \gamma _{2,5}({ \mathbf {S}} \mathbf {B} \mathbf {v} _{r} + \mathbf {w}_{lr})$ ;

24:

${\mathbf z}_{6}^{(n)} \leftarrow {\mathbf z} _{6}^{(n)} + \gamma _{2,6}({ \mathbf {S}} \mathbf {B} \mathbf {v} _{t} + \mathbf {w}_{lt})$ ;

25:

${\mathbf z}_{1}^{(n+1)} \leftarrow {\mathbf z} _{1}^{(n)} - \gamma _{2,1} \mathrm {prox}_{\frac {1}{\gamma _{2,1}}\|\cdot \|_{1,2}}\left({\frac {1}{\gamma _{2,1}} {\mathbf z}_{1}^{(n)}}\right)$ ;

26:

${\mathbf z}_{2}^{(n+1)} \leftarrow {\mathbf z} _{2}^{(n)} - \gamma _{2,2} \mathrm {prox}_{\frac {\lambda }{\gamma _{2,2}}\|\cdot \|_{1,2}}\left({\frac {1}{\gamma _{2,2}} {\mathbf z}_{2}^{(n)}}\right)$ ;

27:

${\mathbf z}_{3}^{(n+1)} \leftarrow {\mathbf z} _{3}^{(n)} - \gamma _{2,3} \mathrm {prox}_{\iota _{B_{p,\alpha }^{\mathbf {0}}}}\left({\frac {1}{\gamma _{2,3}}\mathbf {z}_{3}^{(n)}}\right)$ ;

28:

${\mathbf z}_{4}^{(n+1)} \leftarrow {\mathbf z} _{4}^{(n)} - \gamma _{2,4} \mathrm {prox}_{\iota _{B_{2,\varepsilon _{h}}^{ \mathbf {h}_{r}}}}\left({\frac {1}{\gamma _{2,4}}\mathbf {z}_{4}^{(n)}}\right)$ ;

29:

${\mathbf z}_{5}^{(n+1)} \leftarrow {\mathbf z} _{5}^{(n)} - \gamma _{2,5} \mathrm {prox}_{\iota _{B_{2,\varepsilon _{l}}^{ \mathbf {l}_{r}}}}\left({\frac {1}{\gamma _{2,5}}\mathbf {z}_{5}^{(n)}}\right)$ ;

30:

${\mathbf z}_{6}^{(n+1)} \leftarrow {\mathbf z} _{6}^{(n)} - \gamma _{2,6} \mathrm {prox}_{\iota _{B_{2,\varepsilon _{l}}^{ \mathbf {l}_{t}}}}\left({\frac {1}{\gamma _{2,6}}\mathbf {z}_{6}^{(n)}}\right)$ ;

31:

$n\leftarrow n+1$ ;

32:

end while

D. Detailed Computations and Their Complexity

Table I shows the computational complexity (in $\mathcal {O}$ -notation) of each operation used in Algorithm 1, where the operated vector is $\mathbf {x} = ([\mathbf {x}]_{1}^{\top },\ldots,[\mathbf {x}]_{B}^{\top })^{\top }\in \mathbb {R}^{NB}$ . According to Table I, the computational complexity of each step in one iteration of Algorithm 1 is as follows.

1. Steps 4, 5, 23, and 24: ${\mathcal {O}}(kN_{h}B)$ .

2. Steps 6, 7, 14, 15, 16, 19, 20, 21, 22, 25, 26, 27, and 28: ${\mathcal {O}}(N_{h} B)$ when $p=2$ .

3. Step 9: ${\mathcal {O}}(N_{h})$ .

4. Steps 17, 18, 29, and 30: ${\mathcal {O}}(N_{l} B)$ .

5. Steps 11 and 27: ${\mathcal {O}}(N_{h} B \log (N_{h} B))$ when $p=1$ .

6. Steps 12 and 13: ${\mathcal {O}}(N_{l} B \log (N_{l} B))$ .

After all, one iteration of the algorithm has an overall computational complexity of ${\mathcal {O}}(N_{h} B \log (N_{h} B))$ .

TABLE I Computational Complexity of Each Operation

SECTION IV. Experiments

We demonstrate the effectiveness of our ST fusion method, ROSTF, through comprehensive experiments using simulated and real data for two sites. Our experiments aim to verify the following three items.

1. ROSTF is as effective as state-of-the-art ST fusion methods in noiseless cases and outperforms them in noisy cases. We conducted comparative experiments on four cases of noise contamination. The experimental results for simulated data and real data are presented in Sections IV-D and IV-E, respectively.

2. Key components of ROSTF, such as the assumption-based constraints and the denoising mechanism, operate as expected. To measure their influence, ROSTF without each key component is compared with the original ROSTF in terms of fusion accuracy and convergence speed in Section IV-F.

3. ROSTF is practical in terms of computational time. For a fair comparison, only nondeep-learning-based methods are compared to ROSTF in Section IV-G.

Note that there are two options for ROSTF: $p=1$ or 2, where p corresponds to the choice of the norm in the first constraint in (16), that is, the edge constraint. Hereafter, ROSTF with $p=1$ and ROSTF with $p=2$ are referred to as ROSTF-1 and ROSTF-2, respectively.

A. Data Description

We tested our methods both on real data and simulated data. In the case of satellite observations, radiometric and geometric inconsistencies exist between two different image sensors. This means that the fusion capability of each method cannot be accurately evaluated in experiments using real data because these inconsistencies affect performance, as also addressed in [32]. Therefore, we generated simulated data based on the observation models and verified the performance of each method using this data in addition to the real data. Specifically, in experiments using simulated data, the simulated LR images were generated from the corresponding real HR images according to (14) with $\mathbf {m}=\mathbf {0}$ while the real HR images were used as HR images.

We used MODIS and Landsat time-series images for the following two sites in our experiments.

• Site1:

  The first site is situated in the Daxing district in the south of Beijing city (39.0009° N, 115.0986° E) [16]. For Site1, we employed MODIS and Landsat time-series images acquired on May 29, 2019 (a reference date) and June 14, 2019 (a target date).

• Site2:

  The second site is located in southern New South Wales, Australia (34.0034° S, 145.0675° E) [51]. For Site2, MODIS and Landsat time-series images acquired on January 4, 2002 (a reference date) and February 12, 2002 (a target date) were used.

B. Compared Methods

Our method was compared with STARFM [9], VIPSTF [33], RSFN [28], RobOt [23], and SwinSTFM [31]. Table II shows the characteristics of these methods and ROSTF. Note that, unlike the other methods, RSFN requires input images obtained on two reference dates. Since our experiments assume a scenario with only one reference date, the same HR-LR image pair was input as two different reference image pairs for RSFN.

TABLE II Existing Methods

As the parameters of these existing methods, we used the values recommended in each reference. It should be noted that RSFN and SwinSTFM require significantly more data than our and other existing methods due to training and validation processes. For our experiments, we trained and validated RSFN and SwinSTFM using a different set of images from those used for the tests described in Section IV-A. Specifically, 24 groups from Site1 and two groups from Site2 were used for training, and one group from Site1 and two groups from Site2 were used for validation.

C. Experimental Setup

Our method, ROSTF, is implemented using MATLAB. The source code is available on the GitHub1 platform. For these experiments, the spatial spread transform matrix $\mathbf {B}$ in (14) was implemented as a blurring operator with an averaging filter of size k. The downsampling matrix $\mathbf {S}$ in (14) was designed to take the top-left pixel value in the $k \times k$ window. The parameter k was set to 20 to account for the difference in spatial resolution between the Landsat and MODIS images. The balancing parameter $\lambda $ in (16) was set to 1 since $\widetilde {\mathbf {h}}_{r}$ and $\widetilde {\mathbf {h}}_{t}$ have the same number of elements and are expected to be piecewise smooth to the same degree. The parameters in the constraints in (16) were set as shown in Table III. We set the maximum number of iterations for Algorithm 1 to 50000 and the stopping criterion of Algorithm 1 was set to $\| \widetilde {\mathbf {h}}_{r}^{(n)} - \widetilde {\mathbf {h}}_{r}^{(n-1)}\|_{2}/\| \widetilde {\mathbf {h}}_{r}^{(n-1)} \|_{2}< 10^{-5}$ ,$\| \widetilde {\mathbf {h}}_{t}^{(n)} - \widetilde {\mathbf {h}}_{t}^{(n-1)}\|_{2}/\| \widetilde {\mathbf {h}}_{t}^{(n-1)} \|_{2}< 10^{-5}$ , $\| \mathbf {l}_{r}- \mathbf {S}\mathbf {B} \widetilde {\mathbf {h}}_{r}^{(n)}\|_{2} \leq \varepsilon _{l}$ , and $\| \mathbf {l}_{t}- \mathbf {S}\mathbf {B} \widetilde {\mathbf {h}}_{t}^{(n)}\|_{2} \leq \varepsilon _{l}$ .

TABLE III Parameter Setting

To verify both the pure fusion capability and the robustness against the noise of the existing methods and ours, we conducted the following four combinations of Gaussian noise with different standard deviations and sparse noise (salt-and-pepper noise) with different rates.

• Case1:

  The observed HR and LR images are noiseless, that is, $\sigma _{h}=\sigma _{l}=r_{h}=r_{l}=0$ in (13).

• Case2:

  The observed HR images are contaminated with Gaussian noise with a standard deviation $\sigma _{h}=0.05$ while the observed LR images are noiseless, that is, $\sigma _{h}=0.05, \sigma _{l}=r_{h}=r_{l}=0$ in (13).

• Case3:

  The observed HR images are contaminated with sparse noise with a superimposition rate $r_{h}=0.05$ , while the observed LR images are noiseless, that is, $r_{h}=0.05, \sigma _{h}=\sigma _{l}=r_{l}=0$ in (13).

• Case4:

  The observed HR images are contaminated with Gaussian noise with a standard deviation $\sigma _{h}=0.05$ and sparse noise with a superimposition rate $r_{h}=0.05$ , while the observed LR images are noiseless, that is, $\sigma _{h}=r_{h}=0.05, \sigma _{l}=r_{l}=0$ in (13).

For the quantitative evaluation, we used the following four metrics: the root mean square error (RMSE): \begin{equation*} \mathrm {RMSE} = \sqrt {\frac {1}{N_{h}B} \left \|{ \widetilde {\mathbf {h}}_{t}- \widehat {\mathbf {h}}_{t}}\right \|_{2}^{2}} \tag{21}\end{equation*} where $\widetilde {\mathbf {h}}_{t}$ and $\widehat {\mathbf {h}}_{t}$ denote an estimated HR image and a ground-truth HR image, respectively, on the target date; the spectral angle mapper (SAM) [52] \begin{equation*} \mathrm {SAM} = \frac {1}{N_{h}}\sum _{n=1}^{N_{h}}\mathrm {arccos}\left ({\frac {< \widetilde {\mathbf {e}}_{n},\widehat {\mathbf {e}}_{n}>}{\|\widetilde {\mathbf {e}}_{n}\|_{2} \cdot \left \|{\widehat {\mathbf {e}}_{n}}\right \|_{2}}}\right) \tag{22}\end{equation*} where $\widetilde {\mathbf {e}}_{n}\in \mathbb {R} ^{B}$ and $\widehat {\mathbf {e}}_{n}\in \mathbb {R} ^{B}$ represent the spectral vectors of the nth pixel of $\widetilde {\mathbf {h}}_{t}$ and $\widehat {\mathbf {h}}_{t}$ , respectively; the mean structural similarity overall bands (MSSIM) [53] \begin{equation*} \mathrm {MSSIM} = \frac {1}{B}\sum _{b = 1}^{B}\mathrm {SSIM}\left ({\left [{ \widetilde {\mathbf {h}}_{t}}\right]_{b}, \left [{ \widehat {\mathbf {h}}_{t}}\right]_{b} }\right) \tag{23}\end{equation*} and the correlation coefficient (CC) \begin{equation*} \mathrm {CC} = \frac {s_{ \widetilde {\mathbf {h}}_{t} \widehat {\mathbf {h}}_{t} }}{s_{ \widetilde {\mathbf {h}}_{t}}s_{ \widehat {\mathbf {h}}_{t}}} \tag{24}\end{equation*} where $s_{ \widetilde {\mathbf {h}}_{t} \widehat {\mathbf {h}}_{t} }$ denotes the covariance of $\widetilde {\mathbf {h}}_{t}$ and $\widehat {\mathbf {h}}_{t}$ , and $s_{ \widetilde {\mathbf {h}}_{t}}$ and $s_{ \widehat {\mathbf {h}}_{t}}$ denote the standard deviations of $\widetilde {\mathbf {h}}_{t}$ and $\widehat {\mathbf {h}}_{t}$ , respectively. RMSE was calculated to measure the difference between the estimated image and the ground-truth at the pixel level. SAM was used to measure spectral fidelity. Lower RMSE and SAM values indicate better estimation performance. We used MSSIM to evaluate the similarity of the overall structure. CC shows the strength of the linear relationship between the estimated image and the ground truth. Higher MSSIM and CC values indicate better estimation performance.

D. Experimental Results With Simulated Data

Table IV shows the RMSE, SAM, MSSIM, and CC results in experiments with the simulated data. In Case1, STARFM, VIPSTF, RobOt, SwinSTFM, ROSTF-1, and ROSTF-2 perform equally well. In contrast, the results of RSFN are not good for both the Site1 and Site2 data. This may be because the size of the training data described above was insufficient to effectively train RSFN. Thus, if more training data were used, RSFN might have produced better results. However, collecting noise-free training data is challenging in real-world applications, and this is the scenario considered in these experiments. Next, we focus on the results in Case2, Case3, and Case4, where the observed reference images are contaminated with noise. While STARFM, VIPSTF, RobOt, SwinSTFM, and RSFN demonstrate significantly worse performance due to the influence of noise, ROSTF-1 and ROSTF-2 show no significant performance degradation in Case2, Case3, and Case4. This is because ROSTF estimates the target HR image while simultaneously denoising the reference HR image.

TABLE IV RMSE, SAM, MSSIM, and CC Results in the Experiments With Simulated Data

Figs. 2 and 3 show the estimated results in Case1 for the Site1 and Site2 simulated data, respectively. In the zoomed-in areas, there are significant temporal changes in brightness between the reference HR image $\mathbf {h}_{r}$ and the target HR image, ground truth. For the Site1 data, it is visually apparent that ROSTF-1 and ROSTF-2 capture these changes most accurately and estimate the brightness closest to that of the ground truth compared with the other methods. This is thanks to the brightness constraint and the fifth constraint in (16), which promote the brightness of the estimated image to be close to that of the observed LR image $\mathbf {l}_{t}$ on the target date, based on Assumption 2) and the observation model in (14). Following ROSTF, RobOt also captures the temporal changes effectively. STARFM and VIPSTF are significantly influenced by the reference HR image $\mathbf {h}_{r}$ , resulting in estimates that are closer in brightness to that of $\mathbf {h}_{r}$ rather than to that of the ground truth. The result of RSFN is not good due to a lack of training data. The SwinSTFM result appears to show spectral distortion. For the Site2 data, Fig. 3 illustrates that ROSTF-1 most accurately captures the changes in brightness within the zoomed-in area. ROSTF-2 and RobOt have good estimates, followed by ROSTF-1. STARFM and SwinSTFM do not capture the large temporal changes. The result of VIPSTF has different tints throughout. RSFN estimates brightness closer to that of $\mathbf {h}_{r}$ than to that of the ground truth.

Fig. 2.

ST fusion results for the Site1 simulated data in Case1.

Fig. 3.

ST fusion results for the Site2 simulated data in Case1.

Next, we focus on the results in noisy cases. Fig. 4 shows the estimated results for the Site2 data in noisy cases, that is, Case2, Case3, and Case4. The results of STARFM, VIPSTF, and RobOt are contaminated with a noise similar to that of $\mathbf {h}_{r}$ because they estimate the target image pixel-wise based solely on the noisy reference image $\mathbf {h}_{r}$ . The results of RSFN are corrupted for the unexpected inputs because RSFN was not trained on noisy data. Furthermore, using noisy data to train RSFN may not have produced good results. This is because RSFN is robust to noise only in situations where noisy pixels in the two reference HR images do not appear at the same location. SwinSTFM generates unstable noisy results due to noisy unexpected inputs. On the other hand, ROSTF-1 and ROSTF-2 provide good estimates even when the observed reference images are noisy, without much loss of accuracy compared to the noiseless case. This is thanks to the framework that simultaneously performs noise removal and ST fusion.

Fig. 4.

ST fusion results for the noisy Site2 simulated data. (Top, Middle, and Bottom) Results in Case2, Case3, and Case4, respectively.

The impact of noise on each method is also visually evident in Figs. 5 and 6. The scatter plots in Fig. 5 reveal the difference between the ground-truth and the estimated values of each method for the simulated Site2 data. In Case2, STARFM, VIPSTF, RSFN, RobOt, and SwinSTFM exhibit greater variance compared to Case1 due to the influence of Gaussian noise. In Case3, STARFM, VIPSTF, and RobOt estimate the wrong values close to 0 or 1 affected by sparse noise while the RSFN and SwinSTFM results have no such outliers but show greater variance. Furthermore, in Case4, the distributions indicate that STARFM, VIPSTF, RSFN, RobOt, and SwinSTFM are affected by both Gaussian and sparse noise. In contrast, the results of ROSTF-1 and ROSTF-2 have minimal variance and no outliers, indicating their robustness to Gaussian, sparse, and mixed noise. Spectral profiles of a specific pixel in the results of each method for the Site2 data in Case1 and Case4 are depicted in Fig. 6. STARFM, VIPSTF, RobOt, SwinSTFM, and ROSTF estimate similarly accurate values in Case1, that is, they are comparable in the noiseless case. In Case4, STARFM, VIPSTF, RobOt, and SwinSTFM estimate completely wrong values for the third band affected by sparse noise and perform worse for the other bands due to the influence of Gaussian noise. In contrast, ROSTF-1 and ROSTF-2 have accurate estimates for all bands, even in the noisy case.

Fig. 5.

Scatter plots of the ground truth and the estimated values for the Site2 simulated data.

Fig. 6.

Spectral profiles of a specific pixel in the results of each method for the Site2 data in (a) Case1 and (b) Cas 4.

E. Experimental Results With Real Data

Table V shows the RMSE, SAM, MSSIM, and CC results in experiments with the real data. Compared to the results for the simulated data in Table IV, the performance of ROSTF-1 and ROSTF-2 degrades due to radiometric and geometric inconsistencies between the Landsat and MODIS sensors. Despite the performance degradation, ROSTF-1 and ROSTF-2 perform as well as the existing methods in the noiseless case, that is, Case1, and outperform them in the noisy cases, that is, Case2, Case3, and Case4, as in the experiments with the real data. Thus, it can be concluded that ROSTF is robust to noise even for the simulated data.

TABLE V RMSE, SAM, MSSIM, and CC Results in the Experiments With Real Data

Figs. 7 and 8 show the estimated results in Case1 for the Site1 and Site2 real data, respectively. Compared to the results for the simulated data in Figs. 2 and 3, ROSTF-1 and ROSTF-2 perform worse due to modeling errors between the real HR images $\mathbf {h}_{r}, \mathbf {h}_{t}$ and the real LR images $\mathbf {l}_{r}, \mathbf {l}_{t}$ caused by radiometric and geometric inconsistencies.

Fig. 7.

ST fusion results for the Site1 real data in Case1.

Fig. 8.

ST fusion results for the Site2 real data in Case1.

Fig. 9 shows the estimated results for the Site2 data in the noisy cases, that is, Case2, Case3, and Case4. The results of the existing methods are not good, especially STARFM, VIPSTF, and RobOt generate noisy outputs. The estimated images of ROSTF seem to be blurred in the zoomed area in Fig. 9. This is due to oversmoothing by the HTV regularization terms in (16), which might have undesirable effects on some applications. Nevertheless, according to the difference map in Fig. 10, the results of ROSTF exhibit the least error, and the accuracy evaluation in Table V also shows that ROSTF achieves the best performance in all metrics.

Fig. 9.

ST fusion results for the noisy Site1 simulated data. (Top, Middle, and Bottom) Results in Case2, Case3, and Case4, respectively.

Fig. 10.

Difference map (absolute errors) of the fusion results in the zoomed-in area in the noisy Site1 real data. (Top, Middle, and Bottom) Results in Case2, Case3, and Case4, respectively.

F. Ablation Study

We conducted ablation experiments focusing on the following three items.

1. The edge constraint, $\| \mathbf {D} \widetilde {\mathbf {h}}_{r} - \mathbf {D} \widetilde {\mathbf {h}}_{t} \|_{p} \leq \alpha $ , to encourage similarity in the land structure, specifically the edges, between the reference and target HR images based on Assumption 1).

2. The brightness constraint, $| c_{b}- \mathbf {1}^{\top }[\widetilde {\mathbf {h}}_{t}]_{b}/{N_{h}}| \leq \beta _{b} \, (b = 1, \ldots, B)$ , which is designed based on Assumption 2) to ensure that the estimated target HR image has a similar average brightness to the target LR image.

3. The denoising mechanism for the reference HR image $\mathbf {h}_{r}$ , that is, the first regularization term $\| \mathbf {D} \widetilde {\mathbf {h}}_{r} \|_{1,2}$ and the third, fourth, sixth, and seventh constraints, $\| \mathbf {h}_{r}- (\widetilde {\mathbf {h}}_{r}+ \widetilde {\mathbf {s}}_{hr}) \|_{2}\leq \varepsilon _{h}$ , $\| \mathbf {l}_{r}- ({ \mathbf {S}} \mathbf {B} \widetilde {\mathbf {h}}_{r} + \widetilde {\mathbf {s}}_{lr}) \|_{2}\leq \varepsilon _{l}$ , $\| \widetilde {\mathbf {s}}_{hr} \|_{1} \leq \eta _{h}$ , $\| \widetilde {\mathbf {s}}_{lr} \|_{1} \leq \eta _{l}$ .

We tested ROSTF with each of the three components mentioned above removed. In the following, we present the ablation studies on the two constraints, the edge constraint and the brightness constraint, followed by the ablation study on the denoising mechanism. The hyperparameters in each optimization problem and the stopping criterion of each P-PDS-based algorithm to solve it were set as in the original ROSTF. On the other hand, the stepsizes in each algorithm were set as the values computed according to the operator-norm-based design method of variable-wise diagonal preconditioning in (11).

1) Edge and Brightness Constraints:

First, we measure the effectiveness of the two constraints based on Assumptions 1) and 2), that is, the edge constraint and the brightness constraint, in terms of convergence speed and fusion performance.

Table VI shows the average number of iterations spent before each algorithm stopped and the average performance results for all sites (Site1 or Site2), data types (real or simulated data), and noise cases (from Case1 to Case4). Note that each algorithm always stopped at the maximum number of iterations, 50000, even if the stopping criterion was not met. The original ROSTF performs best with the fewest number of iterations on average, indicating that both of the two constraints contribute to achieving higher fusion performance with fewer iterations.

TABLE VI Average Number of Iterations and the Average Performance Results for All the Situations

Fig. 11 illustrates the transition of the RMSE values for the original ROSTF-2, ROSTF-2 without the edge constraint, and ROSTF-2 without the brightness constraint in the experiment on the Site1 simulated data in Case1. ROSTF-2 without the edge constraint does not meet the stopping criterion until 50000 iterations, possibly because the solution space of the optimization problem without the edge constraint is too large to efficiently reach an optimal solution. This suggests that the edge constraint has the effect of making the solution space moderately small and speeding up convergence. On the other hand, ROSTF-2 without the brightness constraint converges faster than ROSTF-2 without the edge constraint but exhibits unstable behavior, especially in the early iterations. This may be because the variable $\widetilde {\mathbf {h}}_{t}$ is no longer directly updated as a primal variable when the brightness constraint is removed. The update equation corresponding to the brightness constraint is implemented as the only update equation for the primal variable $\widetilde {\mathbf {h}}_{t}$ , as in the lines 8-10 of Algorithm 1, while the update equations corresponding to the other components for $\widetilde {\mathbf {h}}_{t}$ update the dual variables $\mathbf {z}_{2}$ , $\mathbf {z}_{3}$ , and $\mathbf {z}_{6}$ . Directly updating the primal variable leads to faster convergence of the algorithm and more stable behavior than updating the primal variable via the update of the dual variables. Introducing the brightness constraint allows the variable $\widetilde {\mathbf {h}}_{t}$ to be updated directly as the primal variable, thereby enhancing convergence speed and stability.

Fig. 11.

Behavior of the original ROSTF-2, ROSTF-2 without the edge constraint, and ROSTF-2 without the brightness constraint in the experiment on the Site1 simulated data in Case1. The transition of the RMSE values (a) until the algorithms stopped and (b) in early iterations.

Fig. 12 provides a visual comparison of the original ROSTF-2 and ROSTF-2 without these constraints. The image estimated by ROSTF-2 without the edge constraint (a) loses spatial structure, and ROSTF-2 without the brightness constraint and (b) produces an image with incorrect brightness. On the other hand, the original ROSTF-2 (c) estimates brightness close to that of the ground truth while still preserving spatial structure, indicating that both constraints work effectively.

Fig. 12.

ST fusion results of (a) ROSTF-2 without edge constraint, (b) ROSTF-2 without brightness constraint, and (c) original ROSTF-2 for the Site2 simulated data in Case3. (d) Ground truth.

2) Denoising Mechanism:

Next, we move on to the ablation study of the denoising mechanism of ROSTF. The optimization problem of ROSTF without the denoising mechanism is formulated as \begin{align*} \min _{\substack { \widetilde {\mathbf {h}}_{t}, \widetilde {\mathbf {s}}_{lt}}} \: & \|\mathbf {D} \widetilde {\mathbf {h}}_{t}\|_{1,2} \\ \mathrm {s.t.} \: & \begin{cases} \displaystyle \| \mathbf {D} \mathbf {h}_{r} - \mathbf {D} \widetilde {\mathbf {h}}_{t} \|_{p} \leq \alpha, \\ \displaystyle | c_{b}- \mathbf {1}^{\top }\left [{\widetilde {\mathbf {h}}_{t}}\right]_{b}/{N_{h}}| \leq \beta _{b} \, \left ({b = 1, \ldots, B}\right), \\ \displaystyle \| \mathbf {l}_{t}- \left ({{ \mathbf {S}} \mathbf {B} \widetilde {\mathbf {h}}_{t} + \widetilde {\mathbf {s}}_{lt}}\right) \|_{2}\leq \varepsilon _{l},\\ \displaystyle \| \widetilde {\mathbf {s}}_{lt} \|_{1} \leq \eta _{l}. \end{cases} \tag{25}\end{align*} Since the objective function contains only one regularization term for $\widetilde {\mathbf {h}}_{t}$ , no balancing parameter is needed. Also, note that the edge constraint of (25) is different from that of the original ROSTF in (16) because this optimization problem has no variable $\widetilde {\mathbf {h}}_{r}$ .

Table VII displays the average RMSE, SAM, MSSIM, and CC results for each noise case. As expected, in the noisy cases, that is, Case2, Case3, and Case4, ROSTF without the denoising mechanism performs significantly worse than the original ROSTF due to the direct impact of noise. This shows that the denoising mechanism works effectively to make ROSTF robust to noise. On the other hand, we newly found that in the noiseless case, that is, Case1, ROSTF without the denoising mechanism achieves slightly better fusion results than the original one. This result indicates that in noiseless cases, the observed HR image $\mathbf {h}_{r}$ can be used as the reference as it is, and there is no need to estimate $\widetilde {\mathbf {h}}_{r}$ . Furthermore, by removing the variable $\widetilde {\mathbf {h}}_{r}$ , ROSTF does not need the reference LR image $\mathbf {l}_{r}$ as input in (25). Thus, fortunately, if noiseless images can be observed, ROSTF can be applied in situations where only two input images are available, a reference HR image $\mathbf {h}_{r}$ and a target LR image $\mathbf {l}_{t}$ .

TABLE VII Average Performance Results for Each Noise Case in the Ablation Study of the Denoising Mechanism

Fig. 13 illustrates the fusion results of ROSTF-1 without the denoising mechanism for the Site2 simulated data. It is also visually apparent that in Case1, ROSTF-1 without the denoising mechanism produces a satisfactory result using only two input images. On the other hand, in Case2, Case3, and Case4, the results of ROSTF without the denoising mechanism are contaminated with noise. This is because the edge constraint copies not only the true edge or spatial structure, but also the noise in the reference HR image to the estimated target HR image. The results of this ablation study confirm that the denoising mechanism plays an effective role in avoiding such noise effects.

Fig. 13.

ST fusion results of ROSTF-1 without the denoising mechanism for the Site2 simulated data.

G. Computational Cost

We measured the actual running times using MATLAB (R2022b) on a Windows 11 computer equipped with an Intel Core i9-13900 1.0 GHz processor, 32 GB of RAM, and NVIDIA GeForce RTX 4090. We used the Site1 and Site2 data with $400\times 400$ pixels and six bands. The measurement results show that the average number of iterations for Algorithm 1 was 59.20 per second. Table VIII displays the average running times of the nondeep-learning-based comparison methods (STARFM, VIPSTF, and RobOt) and our methods (ROSTF-1 and ROSTF-2). For ROSTF, the average number of iterations for Algorithm 1 is also given in parentheses. Note that only the nondeep-learning-based methods were compared with ROSTF for a fair comparison because deep-learning-based methods require training processes in addition to ST fusion processes.

TABLE VIII Average Running Time [S]

ROSTF-1 and ROSTF-2 each took about 4–10 min. STARFM and VIPSTF took longer than ROSTF because they estimated the target HR image pixel by pixel. On the other hand, RobOt ran much faster than ROSTF, possibly because the Least Absolute Shrinkage and Selection Operator (LASSO) problem in RobOt has a closed-form solution in our experiment with only one reference date. This result indicates that ROSTF is slower than RobOt, but we believe that ROSTF remains practical in terms of computational cost.

H. Summary

We summarize the insights from the experiments as follows.

1. From the results of the experiments in Case1, we see that ROSTF is comparable to state-of-the-art ST fusion methods in noiseless cases. Therefore, the observation model in (14) and the assumptions introduced in Section III-B are valid for ST fusion.

2. The results of the experiments in Case2, Case3, and Case4 confirm that ROSTF has good performance even when observed HR images are degraded by random noise, missing values, and outliers.

3. The ablation studies demonstrate that the key components, such as the edge constraint, the brightness constraint, and the denoising mechanism, work effectively as expected.

SECTION V. Conclusion

We have proposed an optimization-based ST fusion method, named ROSTF, which is robust to mixed Gaussian and sparse noise in observed satellite images. We have formulated the fusion problem as a constrained optimization problem and have developed the optimization algorithm based on P-PDS with OVDP. ROSTF was tested through experiments using both simulated and real data. The experimental results demonstrate that ROSTF achieves performance comparable to state-of-the-art ST fusion methods in noiseless cases and significantly better in noisy cases. ROSTF will have a strong impact on the field of remote sensing, including the estimation of satellite image series with high spatial and temporal resolution from observed image series taken in measurement environments with severe degradation.