In-Network Management of Parallel Data Streams over Programmable Data Planes

Current data centers host an ever increasing number of data analytics applications who are dealing with a growing number of data sources and a continuously increasing volume of data. Parallel stream processing is a powerful paradigm supporting the large-scale deployment of data-analytics applications. However, its performance is limited by its processing capacity of splitting the data streams into parallelizable sub-streams. The splitter that is traditionally executed on general-purpose computational resources can benefit from in-network computing nodes on the communication path. Programmable data planes and corresponding programming models, e.g., Programming Protocol-independent Packet Processors (P4), offer the flexibility of enabling distinct parallelization semantics that can be individually adapted to the dynamic workload. In this paper, we propose Stateful and Scalable Splitter Switch (S4), a network-centric approach leveraging P4 to support parallel stream processing. S4 supports up to 286k concurrent data streams, with a parallelism degree of up to ~ 500k operator instances and a latency overhead of only 2μ
$s$
.