Robust Query Optimization in the Era of Machine Learning: State-of-the-Art and Future Directions

Query optimizers are an essential component of database management systems (DBMSs) as they search for an execution plan that is expected to be optimal for a given query. However, they commonly use parameter estimates that are often inaccurate and make assumptions that may not hold in practice. Consequently, the optimizer may select sub-optimal execution plans at runtime, when these estimates and assumptions are not valid, which may result in poor query performance. Therefore, query optimizers do not adequately support the robustness of the database system. In this tutorial, we explore the notion of robustness in the context of query optimization, as well as how it is evaluated or even further supported. Firstly, we provide a comprehensive definition for the notion of robustness in this context that accounts for risks associated with execution plans and inaccurate parameter estimates as well as the limitations of the cost models. Next, we review the approaches proposed in the literature to address the issue of robustness, including techniques that rely on query re-optimization, discovering parameters, quantifying robustness, as well as recent techniques that employ machine learning. We focus on comparing traditional cost-model-based methods with modern ML-based techniques in terms of their ability to tackle the challenge of robustness in query optimization. Finally, we discuss the limitations and gaps in the current literature and provide some recommendations for future research directions.