Categorising the .mu domain using map-reduce

Mu is the internet country code Top-Level Domain (ccTLD) for the island of Mauritius. It was first introduced in 1995 and is sponsored by the Internet Direct LTD. It is presently administered by the Mauritius Network Information Centre (MUNIC). The domain structure can be broken down into top-level domains (TLD), or first-level domains, and subLevel domains (SLD). The TLD is occasionally viewed as the country code; for example the .mu domain. As Bill Gates said "Domains have and will continue to go up in value faster than any other commodity ever known to man". Undeniably, the .mu domain is currently expanding at an exponential rate as it is not only being regionally accessed. However the main issue with the .mu domain, is the lack of information about the domain itself: its size, types of websites and so on. There is an absence of a directory for the .mu domain. Considering the problems related to the .mu domain name, information retrieval techniques have been used to extract relevant information from the domain. In order to identify and categorise the websites, three techniques have been identified namely: Sequential, Multi-Threading or Map-Reduce. Since we are dealing with large datasets, using traditional computing techniques like the first one won't be appropriate as it is both time and resource consuming. So the best option for this work is the combination of the last two; that is building and testing the URLs in multi-threaded programming and categorise them in the MapReduce framework. The result of our work, is 91.8% of successfully categorized websites out of 1026 websites out of 11,881,376 URLs tested.