Are 3D better than 2D Convolutional Neural Networks for Medical Imaging Semantic Segmentation?

In the last decade, Deep Learning has revolutionized Computer Vision thanks to Convolutional Neural Networks (CNN), that achieved state-of-the-art results in many tasks. In the medical field, imaging techniques, like MRI and CT, are widely used to acquire 3D images of regions that need to be analyzed to identify targets or regions of interest (ROIs). In particular, semantic segmentation is a common image processing task involved in several clinical procedures. When using Deep Learning to solve this task it is possible to either apply a 2D CNN to each slice of the acquired 3D image or apply a 3D CNN to the entire volume acquired. Despite both this approaches have been investigated in the literature, there is neither yet a clear understanding of which one is better (if this is the case) nor a fair comparison of their performances on the same datasets. In this work we aim at making a first step toward to providing an empirical guidance on choosing between 2D and 3D CNNs for medical imaging segmentation. To this purpose we compared a 2D CNN and a 3D CNN based on deep residual U-Net (ResUnet) architecture on different datasets. Our results suggest that the potential benefits of using a 3D CNN are difficult to exploit due to the very limited amount of data that is typically available in medical datasets.