Edge-Guided Dual-Modality Image Reconstruction

To utilize the synergy between computed tomography (CT) and magnetic resonance imaging (MRI) data sets from an object at the same time, an edge-guided dual-modality image reconstruction approach is proposed. The key is to establish a knowledge-based connection between these two data sets for the tight fusion of different imaging modalities. Our scheme consists of four inter-related elements: 1) segmentation; 2) initial guess generation; 3) CT image reconstruction; and 4) MRI image reconstruction. Our experiments show that, aided by the image obtained from one imaging modality, even with highly under-sampled data, we can better reconstruct the image of the other modality. This approach can be potentially useful for a simultaneous CT-MRI system.

SECTION I. Introduction

In general, the imaging modalities are classified into various categories, such as structural imaging (CT/MRI/US) and functional imaging (PET/SPECT). The main advantage of multi-modality imaging is that weaknesses of individual modalities can be offset by the strengths of the others. As a primary example, positron emission tomography (PET) and computed tomography (CT) now work together in the PET-CT scanner for functional and structural studies. This has become the most common tomographic examination in oncological applications. PET-CT not only avoids the inaccuracy from image registration but also saves time and cost for better healthcare. Another multi-modality imaging example is PET-MR, which has been commercially available [1]. PET-MR integrates PET and magnetic resonance imaging (MRI). Despite higher system costs, PET-MR may outperform PET-CT in terms of less ionizing radiation dose and better soft tissue contrast.

Except for PET-CT and PET-MR, there exist other possibilities for multi-modality scanners. CT and MRI are two most popular modalities for structural and functional imaging. CT can provide anatomical structures especially high-contrast details, whereas MRI may depict soft tissues with clarity. Both CT and MRI generate important functional information, aided by classic and novel contrast agents. Clinically speaking, CT and MRI are complementary for evaluation of calcification related diseases, e.g., calcified lesions in the brain [2], [3]. MRI images show the hypo-intense lesions, which are then confirmed by CT images. CT-MRI may be an ideal multi-modality combination for imaging-guided cardiovascular intervention such as heart valve replacement where both anatomical and functional evaluation must be performed in real-time.

In this paper, we investigate the feasibility of unified CT-MRI imaging. When both MRI and CT scans are obtained from an object at the same time, the resultant datasets share inherent similarities (i.e., the boundaries of organs). With an MRI/CT image as the priori information, we could better recover its counterpart image. In the next section, we describe our unified CT-MRI reconstruction method. In the third section, we report our numerical simulation results. Finally, we discuss relevant issues and conclude the paper in the last section.

SECTION II. Method

A. $l_{1}$ -Regularized Problems in Image Reconstruction

In contrast to the Shannon-Nyquist theorem, Candes and Donaho's great work [4], [5] shows that the technique of compressed sensing can accurately recover the signal even if it is sampled at a rather low rate. The success of compressed sensing is based on the assumptions that the signal is sparse (or can be transformed to be sparse), and the measure matrix satisfies the RIP rule. By minimizing the $l_{1}$ -norm of the signal, the solution to the signal recovery problem is almost unique. In reality, most of the signals fulfill the requirement for compressed sensing, and many applications of compressed sensing theory have been demonstrated [6].

The key of compressed sensing is the $l_{1}$ -norm minimization. $l_{1}$ -norm, also known as the Manhattan distance, calculates the sum of absolute component values of a vector $x$ . It is a special case of the $l_{p}$ -norm and expressed as:

$$\begin{equation} \left \| x \right \|_{p} =(\left | {x_{1} } \right |^{p}+\left | {x_{2} } \right |^{p}+\cdots +\left | {x_{n} } \right |^{p})^{\frac {1}{p}}\quad \textrm {with}\quad p=1 \end{equation}$$ Strictly speaking, the $l_{0}$ -norm should be used to help find the sparsest solution from an underdetermined set of equations. However, the $l_{0}$ -norm minimization is an NP-hard problem, and generally too complex to be computationally solved [7]. Fortunately, Candes and Donoho's work suggests clearly that the $l_{1}$ -norm is a good alternative of the $l_{0}$ -norm.

In the field of image reconstruction, the $l_{1}$ -norm minimization is equivalent to reconstruct an image $u$ from few projections (for a CT scan) or partial K-space data (for an MRI scan). A general $l_{1}$ -regularized image reconstruction problem is

$$\begin{equation} {\textrm {arg}\,}\underset {u}{\textrm {min}}\quad \left \| {\Phi u} \right \|_{1} +\frac {\mu }{2}\left \| {Fu-f} \right \|_{2}^{2} \end{equation}$$ where $\mu$ is the relaxation parameter, $f$ is the measured data, $F$ is a transform from the image space to either the sinogram or K-space; that is, the Radon transform for CT imaging or the Fourier transform for MRI imaging, and $\Phi$ is a sparsifying operator upon an image. Since the gradient of the CT/MRI image usually contains many zero or nearly-zero values, formula (2) can be written as $$\begin{equation} {\textrm {arg}\,}\underset {u}{\textrm {min}} \quad \left \| {\nabla u} \right \|_{1} +\frac {\mu }{2}\left \| {Fu-f} \right \|_{2}^{2} \end{equation}$$ This is an unconstraint minimization problem. The $l_{1}$ -regularization term $\left \| {\nabla u} \right \|_{1}$ and the data fidelity $\left \| {Fu-f} \right \|_{2}^{2}$ term are alternatively and iteratively processed to reach a converged result.

There are many papers discussing how to solve the $l_{1}$ -regularized problem. In this work, we use the Split-Bregman technique [8] for MRI reconstruction and the SART-TV algorithm for CT reconstruction respectively. The SART-TV algorithm is very popular in CT image reconstruction; see a detailed description in [9].

The Split-Bregman technique is widely used in recent years, for its high efficiency and accuracy. Let $d=\Phi (u)$ , then formulae (3) can be put in the “Split Bregman” format:

$$\begin{equation} \underset {u,d}{\textrm {min}}\quad \left \| d \right \|_{1} +\frac {\mu }{2}\left \| {Fu-f} \right \|^{2}+\left \| {d-\Phi (u)-b} \right \|^{2} \end{equation}$$ where $b$ is a Bregman parameter.

With $k=0,u^{0}=0,b^{0}=0$ , Eq. (4) is solved iteratively as follows:

$$\begin{align*} &\hspace {-4.8pc} {\textrm {while}}\quad \left \| {u^{k}-u^{k-1}} \right \|_{2}^{2} >\varepsilon ~\textrm {do} \\ u^{k+1}=&\textrm {min}_{u} \left \| {d^{k}-\Phi (u)-b^{k}} \right \|_{2}^{2} +\frac {\mu }{2}\left \| {Fu-f} \right \|_{2}^{2} \\ d^{k+1}=&\textrm {min}_{d} \left \| {d-\Phi (u^{k+1})-b^{k}} \right \|_{2}^{2} +\left \| d \right \|_{1} \\ b^{k+1}=&b^{k}+\Phi (u^{k+1})-d^{k+1} \\ k=&k+1 \\ &\hspace {-4.8pc} \textrm {end while} \end{align*}$$ Clearly, there are two “sub-problems” in the above process. However, the procedure is not as complicated as it appears. The first sub-problem is quadratic in $u$ , and has a closed-form solution with a rather low complexity, while the second sub-problem can be solved using the so-called shrinkage operation [10].

B. Inter-Modality Synergy

In the literature, there has been no ideal solution that transforms an MRI image to a CT image. The CT raw data are collected in the sinogram space, while the MRI raw data are measured in the Fourier space (or K-space equivalently). From the mathematical point of view, the Fourier space and the sinogram space are fundamentally linked, such as by the well-known central slice theorem. However, the physics behind MRI and CT are rather different. MRI relies on the nuclear magnetic resonance of the hydrogen nucleus, while CT targets the attenuation ability of the tissue. Hence, a direct one-to-one mapping is impossible, and an indirect method must be instead used.

In the following, we explain in detail how to generate edge information from an MRI image and how to use it for CT reconstruction, and vice versa. The basic assumption of the proposed method is that the MRI and the CT images are well registered.

1) Segmentation

A generalized $l_{1}$ -norm solver for image reconstruction is first used. The near-zero values are smoothened to keep large values in the gradient image. In most cases, large values correspond to the boundaries of organs, which are connected. A noisy image may have many large values in its gradient image, but they are isolated. From a highly under-sampled dataset, the boundary information (or edge information) may not be fully recovered, thus making the $l_{1}$ -norm solution converge incorrectly. On the other side, if the edge information is pre-defined, the $l_{1}$ -norm solver will be unlikely trapped at a local minimum, yielding the true solution [11].

By inspecting gradient images, it is found that the edge information can be classified into two categories: public and private edges respectively. The public edges are those which can be presented in both imaging modalities, while the private edges are those which are only presented in one imaging modality. Examples of public and private edges are indicated by red and green arrows respectively in Figure 1. To use the edge information, a segmentation procedure is needed. The gradient is a natural segmentation method. An improved method is the Canny edge detector [12]. After image segmentation, the whole image is decomposed into many sub-regions, each labeled a unique value.

FIGURE 1.

Top Row: registered CT and MRI images; Bottom Row: the gradients of the images in the top row.

2) Local Mean

The mean value of each segmented sub-region is calculated by the end of each iteration, and is written back to the image. The local-mean penalty is an extreme case of the $l_{1}$ -regularization. It enforces a reconstructed image to be strictly piecewise constant thereby the $l_{1}$ -norm of the gradient being zero (more generally, piecewise polynominal constraints could be used too). The rationale behind this operation is that during the reconstruction process, low frequency components (regions with slowly changing values) are quickly recovered, and the high frequency components (noise, edges) are gradually recovered. By doing so, public edges are established in the reconstructed image, and the gradient image is much closer to the truth. The next step is to recover private edges of the image. It is achieved by solving the $l_{1}$ -regularization problem defined by Eq. (3), based on the previous result.

SECTION III. Simulation Results

Numerical tests were performed to validate our proposed method. The data were originally synthesized from the NCAT phantom, with some modifications. (http://dmip1.rad.jhmi.edu/~wsegars/phantom_intro.htm).

In the first experiment, we used the CT image as priori information. It was segmented into 9 parts, each was identified by a unique value; see Figure 2. The segmented image was later used by the CSMRI algorithm [8] to generate an initial guess. Only a small area (10% and 15% respectively) of the K-space was sampled in this pilot study. Even for such a simple phantom, the reconstruction was quite challenging since the data were sparsely sampled at random. With the local mean penalty, the initial guess converged very fast, after $3\sim 4$ iterations. Then, the local mean penalty was turned off (or given a small relaxation coefficient), and the CSMRI algorithm was executed with the initial guess until the stopping criteria was satisfied. The results were in Figure 3.

FIGURE 2.

From left to right are the standard CT image, standard MRI image, segmented CT image and segmented MRI image, respectively.

FIGURE 3.

Reconstructed MRI images in the first experiment. Top Row: With 15% of the K-space data; Bottom row: With 10% of the K-space data. Left: Reconstructions using the direct inverse Fourier transform, setting missing samples to zeros in the K-space; middle: Reconstructions using CSMRI; Right: Reconstructions using the proposed method.

A similar process was performed in the second experiment, in which the MRI image was used to guide the CT reconstruction. Fifteen projections were uniformly generated over a $2\pi$ range in the fan-beam geometry. For comparison, we did reconstructions using the SART-TV algorithm with/without the proposed initial guess; see Figure 4. A parameter $\lambda$ was chosen to control the strength of the TV term. The smaller the parameter $\lambda$ is, the stronger the TV term will be. From highly insufficient projection data, the SART-TV algorithm without the initial guess produced severely blocky artifacts ($\lambda =800)$ ; see the left image in Figure 4. Adjusting the parameter $\lambda$ would result in an over-smoothed image ($\lambda =200)$ ; see the middle image in Figure 4. In contrast, a rather large $\lambda$ ($\lambda =2000$ ) was used for the reconstruction with the initial guess since the majority had been restored from the initial guess.

FIGURE 4.

Reconstructed CT images in the second experiment. Left: Reconstruction using SART-TV with $\lambda =800$ ; Middle: Reconstruction using SART-TV with $\lambda =200$ ; Right: Reconstruction using the proposed method.

SECTION IV. Discussions and Conclusion

Compared to CT, MRI is soft tissue sensitive and non-ionizing. However, the scanning time for MRI is much longer than CT because an MRI scanner uses RF pulses to generate MRI signals, and the RF energy deposition is limited by MRI physics. This compromises the dynamic MRI performance. In contrast, CT is very fast. Modern CT acquires thousands projections per second, and advanced CT reconstruction algorithms allow an image to be reconstructed from an incomplete dataset [9]. When MRI is integrated with CT, we hope that the strengths of CT and MRI can be seamlessly integrated.

In this study, we have presented a methodology for CT-MRI imaging. Using an image from one imaging modality as priori information, we can estimate an image of the other modality subject to the local mean penalty. In brief, the missing information in one imaging modality can be effectively compensated for by the priori image from the other image modality, thus relaxing the requirement of individual modality-based measurement significantly for a given image quality. This direction is promising for radiation dose reduction and high-speed MRI in particular.

While we have only investigated the image reconstruction for a CT-MRI combination, a more ambitious goal is towards omni-tomography [13], which is a grand fusion of CT, MRI, PET, SPECT, US, Optical Imaging and more. Each individual imaging modality is only one component of the entire system. In a unified framework, data are collected simultaneously, shared by all the involved modalities, generating more synergistic information on functional, structural, cellular and molecular characteristics of a biological system. How to build closest connections between the images obtained from different modalities and how to fully utilize all prior information will be an interesting topic for our future research.