Gradient-Based Fine-Tuning Strategy for Improved Transfer Learning on Surgical Images

Transfer learning is a widely used technique to leverage pre-trained models on new tasks, but it often suffers from out-of-distribution shifts when the source and target domains are different. This is especially common in surgical images, where the appearance and context of the images vary significantly across different procedures and instruments. To address this problem, we propose a novel gradient-based fine-tuning strategy that selectively freezes layers of a pre-trained model based on their weight gradients. Our method aims to preserve the generalizable features learned from the source domain while adapting the model to the target domain. We evaluate our method on two tasks: surgical instrument recognition and gesture recognition. We compare our method with several existing fine-tuning strategies, including full fine-tuning, linear probing, and gradual unfreezing variations. Our experimental results show that our method achieves the best performance on both tasks. Our approach enables more efficient and robust transfer learning for surgical image segmentation, which is essential for various applications in computer-assisted surgery.