Kafka: the Database Inverted, but Not Garbled or Compromised

The Kafka streaming platform has at its heart a distributed commit log. This resembles the change log that exists in every relational database system. It has been suggested that Kafka be viewed not just as a messaging system, but as the core of a database. The database is in effect “turned inside out” as the normally hidden change log becomes the first class entity of the system, while what is normally considered primary, i.e. the table, view, indexes etc. are just derived from this log. This is appealing as a vision, but raises challenges when applied within an actual enterprise system. The challenges arise from the conflicting interests and requirements of analytics and transactional systems. Running everything on a single system leads to tradeoffs; our intentions here is to identify some of the practical problems with using Kafka as a single data store within an enterprise and to describe our initial approach to resolving them. In particular we present preliminary approaches to ensure consistency and coherence of data from multiple database tables when distributed over Kafka and how to address compliance by encrypting/decrypting data at the Kafka producers and consumers.

I. Introduction

Databases are mutable stores of data allowing changes to data to be made consistently. The store contains a representation of the relational table data and a variety of logs. The terminology and implementation varies from system to system but these commit/transaction/change logs allow auditing, replication and recovery. For example, if a table is to be replicated from one database system to another all the changes to that table can be stored in a change log, such that a remote system can read those changes and replicate them. If all the changes made to a table are stored in a log, then retaining the aggregation of those changes in an explicit table store is (from a theoretically point of view) redundant. Enterprise data systems have evolved over the last fifty years in order to enable high performance. The redundancy of having two representations of data is the price to pay for attaining acceptable response time. The change log representation of a table is normally considered an add-on to the snapshot representation, i.e. they are only enabled when a table must be replicated to another system, but actually it is more fundamental in the sense that the snapshot representation can always be derived from the immutable commit log. The contrary however is not true.