Recovery-Robust Inquiry to External Services from Stream Processing Application

In the commercial development of a stream processing application, it’s frequently required to connect to external existing services or applications from it in order to, for example, retrieve data from a data store server. If we connect them through message queues (MQs) adopting exactly-once (EO) message delivery semantics to avoid duplicate processing of the same messages caused by a failure recovery of the upstream backup technique that involves rereading of messages stored in MQs, it causes serious performance degradation on the inquiries. This becomes a major obstacle in realizing real-time applications.This paper investigates the kind of problems that practically happen in failure recovery of a stream processing application that inquires to external services or applications through MQs adopting at-least-once (ALO) message delivery semantics instead of EO to avoid the latency overhead. The efforts revealed three typical consistency problems on message sequencing that happen in failure recovery. By solving each of the three problems, our developed R³CM technique to inquire to external services/applications from a stream processing application turns the adoption of ALO semantics in MQs into an attractive option to avoid latency overhead.

I. Introduction

Connecting multiple stream processing applications in a message-driven fashion is a basic concept of reactive architecture. As shown in the Reactive Manifesto [1], loose coupling of stream processing applications brings several merits such as isolation of performance and the resilience of the system at a failure. In line with this concept, we aim to connect a real-time stream processing application, which processes arrival messages based on their event time, with another stream processing flow dedicated to inquiries to external services or applications. This is one of common business demands for stream processing applications because they need to, for example, access to external web-services or retrieve data from external database servers.