Tensor Codec: Efficient Differential Codec for real-time AI Inference Communication

With the recent advancements in artificial intelligence (AI) especially deep neural networks, the number of possible use cases and applications has grown exponentially. To realize these use cases and run complex neural networks on a variety of devices (smartphones, wearables, and Internet of Things (IoT) devices, etc.), new technologies such as split computing, distributed computing, federated learning, etc. have come up. These technologies rely on frequently communicating the tensor data generated by complex AI models between devices for performing the inference in the minimum time and with high accuracy. However, the generated tensor data is generally large in size, and communicating the same across devices leads to higher latency, network congestion, and more power consumption. To communicate the tensor data efficiently, we propose a Tensor Codec based on the ISV (Index, Size, and Value) algorithm. The Tensor codec encodes the tensor data efficiently to send only the relevant information between devices using the information of previously sent tensor data. Similarly, on receiving side Tensor codec decodes the encoded tensor data to construct the new tensor data. This not only reduces network congestion but also the latency and power consumption involved in communicating such data. The results show that our proposed solution reduces the data sending by approx. 50%, which in turn decreases the communication latency by approx. 45 to 55%.