M-STREAM: A split model Streaming and Inferencing method for reduced end-to-end Execution Latency

Considering the variety of consumer device portfolios, the use of artificial intelligence (AI) has become an essential tool for its service management and solution acceleration. However, with devices coming in different sizes and processing capabilities, such AI usage sometimes becomes severely constrained. To avoid such constraints, lightweight devices such as augmented reality (AR) glasses, make use of AI-execution offloading solutions to a high-processing capable device or a cloud server. Such solutions introduce a genuine need to transmit the AI model from this lower-capable device to the higher-capable one. Additionally, for additional solutions to enhance the device experience, the service provider frequently updates AI models for next generation multimedia scenarios. This is performed, not only during the establishment of each service session, but even during the media service itself for a better-optimized service, thereby requiring frequent delivery of the newest, or updated version of the AI models from that specific model repository server. Such delivery and download method of such large size models give rise to critical delays in the overall end-to-end execution time, especially for multimedia applications with sensitive MTP (motion-to-photon) and start-time latency requirements. To reduce such latencies, we propose M-STREAM, a split AI model streaming and inferencing method, where delivery and execution can be processed in parallel. The proposed solution shows significant reductions in end-to-end model execution latency maximum upto 50% and on an average around 5% to 25% compared to existing methods.