Query Complexity Based Optimal Processing of Raw Data

Finding a resource-efficient way of processing large application datasets having complex analytical and transactional queries with minimal replication is vital in reducing application running costs. Existing cost-based solutions have high algorithm execution time or data preprocessing requirements. The paper proposes Query Complexity Aware partitioning technique QCA with a lightweight query complexity identification and raw data partitioning algorithm. Different replication approaches have been studied to cover more use cases for different application workloads. The technique is demonstrated using a scientific dataset Sloan Digital Sky Survey SDSS. The results have shown 94.6% reduction in workload execution time WET using only 6.7
^%
of the dataset in loaded format compared to in-situ engine NoDB processing original dataset. QCA technique also reduced multi-node replication by 5.8 times compared to state-of-the-art workload aware WA techniques. The multi-node and multi-core execution of workload using QCA proposed partitions reduced WET by 42.66% and 25.46% compared to WA.