Predicting Web Search Hit Counts

Keyword-based search engines often return an unexpected number of results. Zero hits are naturally undesirable, while too many hits are likely to be overwhelming and of low precision. We present an approach for predicting the number of hits for a given set of query terms. Using word frequencies derived from a large corpus, we construct random samples of combinations of these words as search terms. Then we derive a correlation function between the computed probabilities of search terms and the observed hit counts for them. This regression function is used to predict the hit counts for a user's new searches, with the intention of avoiding information overload. We report the results of experiments with Google, Yahoo! and Bing to validate our methodology. We further investigate the monotonicity of search results for negative search terms by those three search engines.

1. Introduction

Keyword-based search engines often return too many results, which may lead to information overload [1]. Most initial searches produce more than a thousand results, and the hit count estimates of the browsers often report that there are millions of relevant Web pages available. These results are often overwhelming [1]. This does not constitute a problem, as long as the most relevant results are reported on the first few pages of hits. However, often this is not the case. Users often have to sift through many hits to find a few that are relevant, or repeatedly refine their search terms.