Deep Learning-based 3D Magnetic Microrobot Tracking using 2D MR Images

Magnetic resonance imaging (MRI)-guided robots emerged as a promising tool for minimally invasive medical operations. Recently, MRI scanners have been proposed for actuating and localizing magnetic microrobots in the patient’s body using two-dimensional (2D) MR images. However, three-dimensional (3D) magnetic microrobots tracking during motion is still an untackled issue in MRI-powered microrobotics. Here, we present a deep learning-based 3D magnetic microrobot tracking method using 2D MR images during microrobot motion. The proposed method comprises a convolutional neural network (CNN) and complementary particle filter for 3D microrobot tracking. The CNN localizes the microrobot position relative to the 2D MRI slice and classifies the microrobot visibility in the MR images. First, we create an ultrasound (US) imaging-mentored MRI-based microrobot imaging and actuation system to train the CNN. Then, we trained the CNN using the MRI data generated by automated experiments using US image-based visual servoing of a microrobot with a
$500\,{\mu }$
m-diameter magnetic core. We showed that the proposed CNN can localize the microrobot and classified its visibility in an
in vitro
environment with
$\pm \mathbf {0.56}$
mm and
$\mathbf {87.5}$
% accuracy in 2D MR images, respectively. Furthermore, we demonstrated
ex-vivo
3D microrobot tracking with
$\pm \mathbf {1.43}$
mm accuracy, improving tracking accuracy by 60% compared to the previous studies. The presented tracking strategy will enable MRI-powered microrobots to be used in high-precision targeted medical applications in the future.