Dual-Path Model for Pulmonary Artery Segmentation

The pulmonary artery (PA) is a multi-level vascular system composed of the main pulmonary artery (mPA) and the branch pulmonary arteries (bPA). Accurate segmentation of the PA is of significant importance for the diagnosis of diseases such as pulmonary embolism. However, since the vascular features of the mPA and the bPA are very different, the existing holistic PA segmentation methods will cause the network to pay too much attention to the mPA with significant features and ignore the bPA with weak features, resulting in the imbalance of PA segmentation. To address this issue, we propose a Dual-Path Pulmonary Artery Segmentation Model, which employs two separate paths to learn the features of the mPA and bPA, thereby enhancing the network’s ability to learn features of the bPA. Additionally, we have designed a Skeleton-Optimized Feature Learning Mechanism that optimizes the topological structure of the PA through skeletal guidance, reducing fragmentation and false positives. We conducted experiments on the public dataset provided by PARSE2022, and the results demonstrate that our model indeed improves the segmentation accuracy of the bPA while maintaining the segmentation effectiveness of the mPA. Furthermore, the Skeleton-Optimized Feature Learning Mechanism plays a significant role in reducing vascular fragmentation and false positives.

SECTION I. Introduction

The pulmonary artery (PA) is a multi-level vascular system, composed of the main pulmonary artery (mPA) and a multitude of tree-like branches (bPA) which exist within the lungs [1]. PA is associated with numerous diseases, such as pulmonary embolism, pulmonary hypertension, and chronic obstructive pulmonary disease [2]-​[4]. Automating the segmentation of PA in CT pulmonary angiography (CTPA) images is significantly important for clinical diagnosis, lesion localization, and surgical planning.

Deep learning methods are the mainstream approach for PA segmentation, such as [6]-​[10]. Currently, deep learning-based PA segmentation typically segments PA as a whole [11]-​[14]. These studies showed some validity, but they did not take into account the differences in internal features of PA. In the PA, the features of mPA and bPA are unbalanced. The mPA is thick and centralized, and the vascular features are very significant. However, the bPA, which is more important for disease diagnosis [15], is thin and dispersed vessels with weak vascular features. Therefore, existing holistic PA segmentation methods will cause the network to focus excessively on the mPA and ignore the bPA, which will affect the segmentation effect of the bPA. We have proved this phenomenon through relevant experiments, as shown in Fig. 1. As can be seen from the figure, the segmentation effect of bPA in Fig. 1(b) is significantly worse than that of bPA in Fig. 1(c). That is to say, in the experimental setting of holistic PA segmentation, the presence of mPA will lead to inadequate learning of bPA features.

Fig. 1.

(a) PA label. (b) The results of the holistic segmentation of PA using 3D U-Net [5]. (c) The results of segmenting bPA with 3D U-Net. (d) The results of segmenting mPA with 3D U-Net.

Fig. 2.

The overview of Dual-Path Pulmonary Artery Segmentation Model.

To solve the above problem, we propose the Dual-Path Pulmonary Artery Segmentation Model (DP-PASM), which leverages the Dual-Path mechanism to enable independent learning of the mPA and bPA, thereby enhancing the segmentation accuracy of the bPA while ensuring the segmentation effectiveness of the mPA. DP-PASM employs two paths to extract features of the mPA and bPA, respectively. The mPA-Path employs a lightweight 4-layer U-shaped network to learn the features of the mPA. And a subtle feature enhanced U-shape network is used for bPA-Path to extract more subtle features of the bPA. Subsequently, a fusion inference module is utilized to integrate the outputs from both paths, and the complete PA segmentation result is obtained. Additionally, to further optimize the topology of PA, we propose a Skeleton-Optimized Feature Learning Mechanism (SO-FLM), which extracts the skeletons of both the annotated and network-predicted vessels, allowing the predicted vessel skeletons to learn from the annotated ones. This process optimizes the topological structure of the predicted vessels, reducing fragmentation and false positives.

The contributions of this paper are summarized as follows:

• We proposed a Dual-Path Pulmonary Artery Segmentation Model, which employs two paths to learn the features of mPA and bPA, respectively, thereby mitigating the impact of mPA on bPA;

• We proposed a Skeleton-Optimized Feature Learning Mechanism to optimize the topological structure of PA, reducing fragmentation and false positives;

• Extensive experiments are conducted on the PARSE2022 dataset [1] demonstrate that is of great significance in improving the segmentation effect of bPA and optimizing the topological structure of PA.

SECTION II. METHOD

Due to the significant differences in vascular features between the mPA and bPA, segmenting the holistic PA can lead the network to focus excessively on the mPA, which has significant vascular features while ignoring the bPA with weak vascular features. To address this problem, we propose the DP-PASM, which employs two paths to learn the features of the mPA and bPA, respectively.

A. Architecture

The DP-PASM consists of an Image Decomposition Module, an mPA-Path Module, a bPA-Path Module, and a Fusion Inference Module, with the overall architecture as shown in Fig. 2.

In the Image Decomposition Module, the intact PA image is decomposed into mPA and bPA images. We initially extract the lung mask Mask_lung from the CTPA images by employing lung segmentation [16] and the largest connected component operation. Subsequently, the images and labels are cropped based on Mask_lung to minimize interference from irrelevant areas, resulting in Image_PA and Label_PA. Thereafter, the intersection of Image_PA and Mask_lung is saved as bPA images $Image_{bPA}^\prime$, while the set difference is saved as mPA images $\operatorname{Im} age_{mPA}^\prime$. The specific process is as follows: $$\begin{align*}&Image_{bPA}^\prime = Imag{e_{PA}}\mathop \cap \nolimits^ Mas{k_{lung}},\tag{1a} \\ & \operatorname{Im} age_{mPA}^\prime = \operatorname{Im} ag{e_{PA}} - Mas{k_{lung}}\tag{1b}\end{align*}$$

Label_PA is decomposed into Label_bPA and Label_mPA in the same manner. Finally, the intensities of $\operatorname{Im} age_{mPA}^\prime$ and $Image_{bPA}^\prime$ are clipped to [50, 350] HU and [-850,50] HU, respectively, to obtain Image_mPA and Image_bPA.

The mPA-Path Module takes Image_mPA as input and outputs the segmentation result for mPA Pre_mPA. Given the strong vascular features of the mPA, a simple network can achieve excellent segmentation results. Therefore, the mPA-Path Module employs a lightweight 4-layer U-shape network [17]. The 4-layer design not only ensures effective feature learning for mPA but also improves the memory usage efficiency.

The bPA-Path Module takes Image_bPA as input and outputs the segmentation result for bPA Pre_mPA. In contrast to mPA, the bPA has a more complex topological structure and weaker vascular features. Therefore, a deeper subtle feature enhanced U-shape network is used for the bPA-Path Module to extract more refined features. We will provide a detailed description of the bPA-Path Module in Section II-B.

In the Fusion Inference Module, Pre_mPA and Pre_bPA learned by the two path modules are fused to obtain the final complete PA segmentation result Pre_PA. The process is specifically represented as: $$\begin{align*}&Pre_{PA}^\prime = concatenate\left({{\text{Pr}}{{\text{e}}_{mPA}},Pr{e_{bPA}}}\right),\tag{2a} \\ & Pr{e_{PA}} = 1 \times 1 \times 1Conv3d\left({{\text{Pre}}_{PA}^\prime }\right).\tag{2b}\end{align*}$$

Additionally, since the networks in the path modules focus only on the detailed features of the vessels and neglect the overall structural features, we incorporate the SO-FLM into the Fusion Inference Module to optimize the topological structure of the PA.

B. Residual-Based Subtle Feature Enhancement Mechanism for the bPA-Path Module

The bPA-Path Module also employs a U-shape network to extract features of the bPA. However, due to the complex topological structure of the bPA, a deeper 5-layer U-shape network is utilized. Additionally, the vascular features of the bPA are subtle which is easy to be ignored in the feature learning process, resulting in the loss of features. Therefore, a Residual-Based Subtle Feature Enhancement Mechanism (RB-SFEM) is proposed to enable the network to retain subtle features during feature learning, avoid feature loss, and enhance the network’s learning of bPA features.

The network consists of two stages: down-sampling and up-sampling. During the down-sampling stage, max-pooling easily leads to the loss of subtle features in the input image. To preserve the subtle vascular features during down-sampling and prevent feature loss, the RB-SFEM is used before and after max-pooling to enhance the subtle features in the input image. RB-SFEM uses residual connections [18] to strengthen the subtle vascular features, thereby enhancing the salience of the features, which can be specifically represented as: $$\begin{align*}&F_b^1(x) = ReLU(BN(Conv3d(x))),\tag{3a} \\ & F_b^2(x) = BN\left({Conv3d\left({F_b^1(x)}\right)}\right),\tag{3b} \\ & {F_b}(x) = \operatorname{Re} LU\left({F_b^2(x) + x}\right),\tag{3c}\end{align*}$$ where x, F_b(x) represents the input and output of RB-SFEM, and BN denotes BatchNorm3d. During the up-sampling stage, ConvTranspose can lead to the loss of learned subtle features. Therefore, RB-SFEM is also used before and after ConvTranspose to enhance the subtle features, ensuring that the network can propagate the learned subtle features to Pre_bPA.

C. Skeleton-Optimized Feature Learning Mechanism

The path modules focus only on the detailed features of the vessels, neglecting the overall structural features. It leads to discontinuities and false positives in the PA segmentation results. To optimize the topological structure of the vessels, SO-FLM is employed in the Fusion Inference Module to add constraints on the vascular topology.

Fig. 3.

SO-FLM schematic diagram.

SO-FLM, as depicted in Fig. 3, employs morphological operations to extract the skeleton of the blood vessels. In the implementation, the initial skel is set to be empty, and the following process is iteratively performed: $$\begin{align*}&img = erode(img),\tag{4a} \\ & O(img) = open(img),\tag{4b} \\ & {S^\prime } = img - O(img),\tag{4c} \\ & skel = skel + {S^\prime }.\tag{4d}\end{align*}$$

erode and open represent the erosion and opening operations, respectively. skelrepresents the final extracted skeleton. The SO-FLM extracts the skeletons Skel_Pre and Skel_Label from the network’s predictions and the labels, respectively. The topology of the PA is optimized through feature learning between these skeletons.

The SO-FLM is concretely implemented within the model as a skeleton-guided loss function, Skel-Dice loss. This function is defined as the Dice loss between Skel_Pre and Skel_Label, and is specifically expressed as: $$\begin{equation*}Los{s_{Skel}} = 1 - \frac{{2 \times \left({Ske{l_{\Pr e}} \cap Ske{l_{Label}}}\right)}}{{{}Ske{l_{\Pr e}} \cup Ske{l_{Label}}}}.\tag{5}\end{equation*}$$

The total loss of the model is calculated as $$\begin{equation*}Loss = \alpha \cdot Los{s_{mPA}} + \beta \cdot Los{s_{bPA}} + \gamma \cdot Los{s_{PA}}.\tag{6}\end{equation*}$$

Here, α, β and γ represent hyper-parameters that balance the three losses. The Loss_n, where n ∈ {mPA,bPA,PA}, is computed as follows: $$\begin{equation*}Los{s_n} = {\alpha ^\prime } \cdot Los{s_{CE}} + {\beta ^\prime } \cdot Los{s_{Dice}} + {\gamma ^\prime } \cdot Los{s_{Skel}}\tag{7}\end{equation*}$$

Loss_CE, Loss_Dice, and Loss_Skel denote the Cross-Entropy loss, Dice loss, and Skel-Dice loss, respectively.

SECTION III. Experiments

A. Data and Experimental Setup

Experiments were conducted using the public dataset provided by PARSE2022 [1]. The dataset comprises 100 CTPA images along with their PA labels. We randomly allocated 80% for training and 20% for testing the model. Due to GPU memory limitations, images were divided into patches of size 96×96×96 for training and testing to reduce memory consumption.

We evaluated the segmentation accuracy of the model using the Dice and HD95 metrics. Considering the varying difficulty of segmenting mPA and bPA, we adopted two-level Dice and HD95 similar to PARSE2022, assigning 20% and 80% weights to mPA and bPA, respectively. The final weighted score is defined as the ultimate score for the PA.

TABLE I The comparison among DP-PASM and other models. PARSE2022 T1-T5 denotes the models utilized by the first to fifth place winners of the PARSE2022 challenge [1]. It should be noted that the competition also evaluates models based on time and memory consumption, hence the ranking order does not strictly equate to the order of segmentation metrics.

All experiments were performed on four 3090 GPUs, each with 24G of memory. The Adam optimizer was utilized with an initial learning rate of 0.001, and the model was trained for a total of 250 epochs.

B. Pulmonary Artery Segmentation Results

The performance of DP-PASM was compared with the competition results of PARSE2022 and several other methods. The results are summarized in Table I. It can be observed from the table that DP-PASM outperforms all the top 5 models from the PARSE2022 challenge, both in terms of individual results for mPA and bPA, as well as their weighted combination.

Furthermore, compared to the baseline 3D U-Net, although our results were slightly worse than the results of 3D U-Net in the evaluation of mPA, DP-PASM has greatly improved the segmentation effect of bPA, which is more difficult to segment: increasing the Dice score by 7% and reducing the HD95 result by 6mm. We think the sacrifice is worth it. And from the final weighted results, DP-PASM is still far superior to all other models.

C. Ablation Study

In order to verify the effectiveness of Dual-Path, we conducted relevant ablation experiments. We compared the results of using a single path to segment holistic PA images with the results of using Dual-Path on mPA and bPA images, and index scores are shown in Table II. From the table, it can be observed that when the Dual-Path model is not used and a single path is directly used to segment holistic PA, the dice score for bPA is only 80.08%, and HD95 is 5.73mm. However, when the Dual-Path model is employed, although there is a slight decrease in the performance for mPA, the dice score for the more critical bPA increases to 81.30%, and HD95 is reduced to 3.86mm. This indicates that the Dual-Path model can further enhance the segmentation effect of bPA while ensuring the segmentation effect of mPA, and the overall segmentation accuracy of PA has also been improved.

TABLE II The results of ablation experiments. The first two lines represent that a single path is used to segment PA on the complete PA image, while the third line indicates that Dual-Path architecture is used to segment PA in the mPA and bPA images.

Fig. 4.

PA segmentation results before and after SO-FLM was added. The yellow area indicates the false positive area, and the blue area indicates the false negative area.

What’s more, we also conducted ablation studies on SOFLM. Fig. 4 shows the segmentation results of the PA before and after incorporating SO-FLM. The green boxes highlight the changes in the segmentation results with the addition of SO-FLM. It can be observed that SO-FLM does indeed improve the segmentation of PA to some extent, particularly in addressing PA fragmentation and false positives.

SECTION IV. CONCLUSION

Due to the significant differences in vascular features between the mPA and bPA, existing holistic PA segmentation methods may cause the network to focus excessively on the mPA while ignoring the bPA with subtle vascular features. To address this issue, we propose the DP-PASM, which employs two paths to learn the features of the mPA and bPA in parallel, thereby mitigating the impact of the mPA on the bPA. Additionally, we propose the SO-FLM to optimize the topological structure of the PA, reducing fragmentation and false positives. Experiments conducted on the PARSE2022 dataset demonstrate that DP-PASM outperforms other models and indeed enhances the segmentation accuracy of the bPA while maintaining the segmentation effectiveness of the mPA. Furthermore, the results indicate that SO-FLM plays a significant role in reducing vascular fragmentation and false positives.