Speech-image based Multimodal AI Interaction for Scrub Nurse Assistance in the Operating Room

With the increasing surgical need in our aging society, there is a lack of experienced surgical assistants, such as scrub nurses. To facilitate the training of junior scrub nurses and to reduce human errors, e.g., missing surgical items, we develop a speech-image based multimodal AI framework to assist scrub nurses in the operating room. The proposed framework allows real-time instrument type identification and instance detection, which enables junior scrub nurses to become more familiar with the surgical instruments and guides them throughout the surgical procedure. We construct an ex-vivo video-assisted thorascopic surgery dataset and benchmark it on common object detection models, reaching an average precision of 98.5% and an average recall of 98.9% on the state-of-the-art YOLO-v7. Additionally, we implement an oriented bounding box version of YOLO-v7 to address the undesired bounding box suppression in instrument crossing over. By achieving an average precision of 95.6% and an average recall of 97.4%, we improve the average recall by up to 9.2% compared to the previous oriented bounding box version of YOLO-v5. To minimize distraction during surgery, we adopt a deep learning-based automatic speech recognition model to allow surgeons to concentrate on the procedure. Our physical demonstration substantiates the feasibility of the proposed framework in providing real-time guidance and assistance for scrub nurses.