Prediction of Web Page Relevance in Focused Crawling Using Artificial Neural Networks

In the rapidly expanding digital ecosystem, efficient web crawling is paramount to extract relevant information from the vast expanse of the internet. This research addresses the challenge of predicting web page relevance in focused crawling using Artificial Neural Networks (ANNs). The study leverages a curated dataset of web pages, annotated for their relevance to specific domains. A multi-layer perceptron architecture was employed for the ANN, with features extracted from the web pages serving as inputs. The model was trained and validated using a split of 70% training and 30% testing data. Experimental results indicate that the ANN achieved an accuracy of 93.20%, outperforming traditional methods by a margin of 8%. This demonstrates the potential of ANNs in enhancing the efficiency of focused web crawlers, ensuring more accurate data extraction in less time. The findings pave the way for integrating deep learning techniques into web crawling systems, optimizing them for the evolving digital landscape.