Accelerate Large Language Model Inference on Edge TPU with OpenVX framework

This paper proposes a novel method to accelerate inference of Large Language Models (LLMs) on Edge Tensor Processing Units (TPUs) by leveraging the OpenVX framework. The approach harnesses the synergistic integration of OpenVX with Edge TPUs to enable parallel processing capabilities and address precision challenges in performing convolutions exclusively through integer arithmetic. Since LLMs, unlike Convolutional Neural Networks (CNNs), cannot automatically construct network architectures via Open Neural Network Exchange (ONNX) parsing, manual network construction using the OpenVX API is necessitated. This demands precise understanding of each constituent component of the original model, as any inaccuracies propagate to entirely erroneous results. Through quantization, the proposed technique preserves accuracy during convolution operations while reusing the integer arithmetic convolution capabilities of Edge TPUs for efficient inference. The method showcases the collaborative synergy achieved between OpenVX and Edge TPUs, enhancing efficiency of LLM inference on resource-constrained edge devices.