DOLL: Distributed OnLine Learning Using Preemptible Cloud Instances

To defray the increasingly massive costs of running large machine learning workloads, much work has proposed running them on preemptible cloud instances, a discount tier of virtual machine rentals that may be interrupted at the cloud provider's discretion. This work, however, largely ignores the fact that much data used for machine learning comes from streams of diverse sources, e.g., wirelessly connected cameras or hospital health records. Processing datastreams on preemptible instances presents new challenges: processing data as they arrive may engender bottlenecks when scaling the system to handle higher throughput, particularly if the instances are frequently interrupted. Ours is the first work to design, analyze, and optimize a system that uses a set of datastreams to train a machine learning model on preemptible instances. Our system, DOLL, uses queueing and batching to parallelize and scale SGD (stochastic gradient descent)-based optimizers to large numbers of workers and datastreams, as well as heterogeneous data arrival rates across streams. Expected error convergence guarantees are then derived for DOLL's training process. We use this guarantee to optimize the cost of requisitioning preemptible and on-demand instances given an error target and wall-clock time deadline; this optimization is validated on experiments demonstrating substantial cost savings with little impact on model error.