FLEX: Adaptive Task Batch Scheduling with Elastic Fusion in Multi-Modal Multi-View Machine Perception

This paper presents FLEX, a real-time scheduling framework that adaptively allocates limited machine attention (i.e., computing resources) among different spatial views (partitioned by camera facing directions) and sensory modalities (i.e., LiDAR and cameras) within multi-modal multi-view machine perception on resource-constrained embedded platforms. It is achieved through the effective wiring of two features: First, considering the heterogeneous and time-varying criticality among views and modalities within dynamic sensing contexts (i.e., object locations), we calibrate an “anytime” multi-modal perception pipeline that dynamically adjusts the modality fusion strategies of each view. Second, to optimize the GPU processing throughput with time guarantees, FLEX centers around an adaptive batch scheduling algorithm that intelligently groups consecutive asynchronous view inspection tasks based on the job sequence¹ generated from a non-preemptive EDF schedule to maximize a measure of system utility, with the runtime elastic fusion used as a subroutine. Temporal load balancing is maintained during the scheduling by always ensuring the sequential schedulability of future tasks in early batching decisions. We implement FLEX on NVIDIA Jetson Orin and conduct extensive experiments with a large-scale driving dataset. The results demonstrate the superiority of FLEX in improving perception quality and system throughput with time guarantees.¹Following the real-time conventions, we use “job” to denote the instance of periodic tasks.

I. Introduction

Machine perception based on deep neural networks (DNNs) is widely deployed in autonomous systems such as autonomous vehicles (AV) and unmanned aerial vehicles (UAV). They are typically equipped with multiple types of sensory modalities (e.g., RGB cameras and LiDAR), and rely on a 3D object detector integrating information from all modalities and different views to localize and classify surrounding objects [1] –[3]. It is imperative to achieve accurate environment perception while keeping up with the real-time requirement, where the trade-off becomes more complex within dynamically evolving physical environments.