Indexing process insight and evaluation

Indexing is an important process in Information Retrieval (IR) systems. It forms the core functionality of the IR process since it is the first step in IR and assists in efficient information retrieval. Indexing reduces the documents to the informative terms contained in them. It provides a mapping from the terms to the respective documents containing them. Once effective index has been built for the collection of documents, retrieval process is simplified. Indexing proceeds at four stages namely content specification, tokenization of documents, processing of document terms, and index building. The index can be stored in the form of different data structures namely direct index, document index, lexicon and inverted index. Index can be built by applying different algorithms or schemes such as single-pass in-memory indexing, blocked-indexing, etc. This paper explains the indexing process with the various data structures and algorithms used for indexing and finally analyses the different indexing approaches with respect to time, memory usage and mean average precision.

I. Introduction

An Information Retrieval system comprises of three core functionalities: indexing, retrieval and query analysis. Indexing is the process that manages the content [1] or collection for efficient retrieval. Indexing is carried out by the indexer module [2]. It was found earlier in the history of IR that to avoid linear scanning, indexing of documents in advance is required. In indexing, it is assumed that both the documents and the user query are expressed as a set of terms (index terms). Retrieval on the other hand, performs the matching of the document and the query terms to retrieve the relevant documents which are expected to be of informative value to the user. Indexing, thus, is intended towards simplifying the retrieval process.