Edge-Optimized Model for Multimedia Classification using Linguistic Metadata

Language models are relevant for text analysis. Transfer learning enables fine-tuning pre-trained large-language model (LLM) architectures for various classification and prediction tasks. However, these fine-tuned LLMs are computationally intensive, have large memory requirements, and have high inference latency, as shown in this paper, which can prevent the deployment of such models for real-time applications on edge devices. This paper presents results from a joint optimization between a low-rank factorization of a text embedding model and a recurrent long short-term memory (LSTM) model using linguistic metadata for a seventeen-class multimedia classification problem. A comparative study shows that our approach exceeds the performance of state-of-the-art large-language models in latency and number of parameters while performing approximately with the same accuracy as larger models, enabling real-time inference on an edge device. Consequently, the model performs real-time inference on a consumer TV for multimedia classification.