A Teacher-Student Framework For Training Audio Spectrogram Transformer To Classify Lung Diseases Using Respiratory Sounds

In this study, we explore the potential of utilizing transformers in a limited data setting. We propose a teacher-student framework to train transformers for classifying lung diseases using respiratory sounds. Our proposed framework combines the attributes of both Convolutional Neural Network (CNN) and Transformer i.e., the translation equivariance and inductive biases of CNNs with the ability of transformers to handle long range dependencies. We have used Wavegram- Logmel-CNN as the teacher and Audio Spectrogram Transformer (AST) as the student model. The results show that our proposed framework improves the accuracy of the Transformer model. We also discuss the future scope for further improvements.