Big Data Management System Architectures: From Opportunities to Challenges [Vision]

Over the years, data management systems have undergone significant transformations adapting to advancements in hardware, diverse data types, and varying workloads. Numerous architectures have been proposed to more effectively address the customer needs, taking into account rapidly evolving factors like emerging data types, shifting workload and access patterns, and modern hardware innovations. With emergence of cloud infrastructures and their improved capability in seamless scaling, the shared-nothing architecture, which dominated the field for about three decades, is now being replaced with disaggregated architectures. The considerable cost of cloud resources has significantly increased the interest in further disaggregating resources, enabling them to scale independently of one another as required. Consequently, cost-sharing via multi-tenancy and the benefits of pay-for-use billing policies in serverless computing have become pivotal in shaping the design of new database architectures. This paper provides a concise overview of database architecture evolution and their triggers for change, the contemporary state-of-the-art structures, and highlights open research problems for future exploration.