Instructional Activity Recognition Using A Transformer Network with Multi-Semantic Attention

Instructional activity recognition is an analytical tool for the observation of classroom education. One of the primary challenges in this domain is dealing with the intricate and heterogeneous interactions between teachers, students, and instructional objects. To address these complex dynamics, we present an innovative activity recognition pipeline designed explicitly for instructional videos, leveraging a multi-semantic attention mechanism. Our novel pipeline uses a transformer network that incorporates several types of instructional semantic attention, including teacher-to-students, students-to-students, teacher-to-object, and students-to-object relationships. This comprehensive approach allows us to classify various interactive activity labels effectively. The effectiveness of our proposed algorithm is demonstrated through its evaluation on our annotated instructional activity dataset.