OpenStack in Higher Education and Academic Research: A Case Study on Benchmarking Big Data Processing Tools

Despite many generous grants and discounts provided by major suppliers of cloud resources for academia and/or research teams, the cost incurred by the deployment and operationalization of cloud storage and processing is far from trivial. Hybrid cloud open-source solutions such as OpenStack can prove extremely important for institutions lacking generous financial resources. This is the case with the RaaS-IS platform installed at Alexandru Ioan Cuza University of Iasi which hosts a large variety of research projects on Big Data, Machine Learning, Text Mining, Molecular Biology, etc. developed by research teams from various institutions. This paper introduces a technical solution built on OpenStack, designed to assess the performance of data processing—specifically SQL queries—across a broad spectrum of Big Data tools, such as Apache Hadoop and Spark. The aim of the solution is to evaluate SQL query performance as data volume scales from hundreds of gigabytes to terabytes. The evaluation considers various query parameters, including the number of joins specified in the FROM clause, predicate filters in the WHERE clause, attributes in GROUP BY, conditions in the HAVING clause and many more.