An adaptive and efficient dimensionality reduction algorithm for high-dimensional indexing

The notorious "dimensionality curse" is a well-known phenomenon for any multidimensional indexes attempting to scale up to high dimensions. One well known approach to overcoming degradation in performance with respect to increasing dimensions is to reduce the dimensionality of the original dataset before constructing the index. However, identifying the correlation among the dimensions and effectively reducing them is a challenging task. We present an adaptive multilevel mahalanobis-based dimensionality reduction (MMDR) technique for high-dimensional indexing. Our MMDR technique has three notable features compared to existing methods. First, it discovers elliptical clusters using only the low-dimensional subspaces. Second, data points in the different axis systems are indexed using a single B/sup +/-tree. Third, our technique is highly scalable in terms of data size and dimensionality. An extensive performance study using both real and synthetic datasets was conducted, and the results show that our technique not only achieves higher precision, but also enables queries to be processed efficiently.

1 Introduction

Many database applications, such as multimedia retrieval, exploratory data analysis, market basket application and time-series matching, involve high-dimensional data. Indexing high-dimensional data has been an area of active research for a long time and many indexing techniques have been proposed [13]. However, the performance of these indexes degrades rapidly with increasing dimensionality [3].