Topic Based Partitioning for Selective Search Using Sharding Technique

In today's technological revolution many IT industries, government and private organisations are generating huge volume of data every day. To process this data distributed architecture systems are used. The major challenges in processing data in distributed systems are scalability, reliability, availability and portability. The traditional search techniques used for searching and processing data in distributed systems have high latency and less throughput. As a case study Elasticsearch is used to store and process data. Elastic search uses distributed architecture where Master node is connected to multiple slave nodes or data nodes. The main goal of this paper is to efficiently store and process data in elasticsearch using sharding technique. Sharding is a horizontal data partitioning technique which partitions the data into multiple data nodes. In Existing search operation all the data nodes were searched to retrieve relevant data matching to the user query. Using sharding technique we will search only selected shards to retrieve relevant data this reduces the latency and enhances the throughput, scalability and performance of distributed processing system. Results interpret that using sharding techniques the throughput and performance has enhanced. A comparative study analysis of different shard selection technique is carried out considering precision and similarity score metrics and results infer that ReDDe shard selection algorithm is efficient compared to exhaustive and CORI techniques.