DIME: Distributed Inference Model Estimation for Minimizing Profiled Latency

Distributed inference allows minimizing metrics such as latency by offloading some computations from an edge device. It is commonly formulated and solved as an Integer Linear Program (ILP) for layer-wise partitioning of a Deep Neural Network (DNN) to decide transition points from an edge device to a hub and/or cloud devices. The formulation requires parameters reflecting latencies to execute different bundles of consecutive layers of DNN on each device. Profiling is the main way to measure these bundle latencies accurately on a device. In this work, we show a recent ILP of the layer-wise partitioning (JointDNN) cannot in fact always generate an optimal solution. As we show, this happens due to profiling behavior seen in some devices. We propose DIME (Distributed Inference Model Estimation) with novel modifications to accurately estimate the latency of a bundle within the ILP formulation. It guarantees generating the optimal solution regardless of the type of device in the network. Additionally, DIME incorporates a new input parameter within the ILP to control the tradeoff between solution quality and the profiling effort. In our experiments we show solving DIME always results in the optimal solution, sometimes with significantly less profiling effort.