Enhancing Transfer Learning Reliability via Block-Wise Fine-Tuning

Fine-tuning can be used to tackle domain specific tasks by transferring knowledge learned from pre-trained models. However, previous studies on fine-tuning focused on adapting only the weights of a task-specific classifier or reoptimising all layers of the pre-trained model using the new task data. The first type of method cannot mitigate the mismatch between a pre-trained model and the new task data, and the second type of method easily causes over-fitting when processing tasks with limited data. To explore the effectiveness of fine-tuning, we propose a novel block-wise optimisation mechanism, which adapts the weights of a group of layers of a pre-trained model. This work presents a theoretical framework and empirical evaluation of block-wise fine-tuning to find a reliable transfer learning strategy. The proposed approach is evaluated on two datasets, Oxford Flowers and Caltech 101, using 15 commonly used state-of-the-art pre-trained base models. Results indicate that the proposed strategy consistently outper-forms the baselines in terms of classification accuracy, although the specific block leading to optimal performance may vary across models. The investigation reveals that selecting a block from the fourth quarter of a base model generally yields improved performance compared to the baselines. Overall, the block-wise approach consistently outperforms the baselines and exhibits higher accuracy and reliability. This study provides valuable insights into the selection of salient blocks and highlights the effectiveness of block-wise fine-tuning in achieving improved classification accuracy in various models and datasets.