Topic Clustering Using Induced Squared Correlation Thresholding with Dimension Reduction

Clustering algorithms are now increasingly being adopted for the detection of bursty events in Online Social Networks (OSNs). Traditionally designed to be used predominantly with structured dataset as an unsupervised technique, it is gaining popularity for text analytics in recent years. In a bid to achieve better accuracy when applying cluster analysis, variants of the original clustering algorithm have emerged within the research community and there is still no end in sight. The success of the clustering algorithms largely depends on the ideal cluster number that is initially set as a parameter before running the cluster analysis. Initial analysis, in addition to the selection of useful features, usually precedes the actual cluster analysis, which can provide useful insights whilst setting up the optimal number of clusters when analysing structured datasets. In the context of event detection, this translates to the number of events in an unstructured data. Using cluster analysis to infer the number of events in a text document may be less informative if the number of clusters is not carefully chosen, which also can make the entire process tedious and complex. With this in mind, this paper proposes an efficient strategy of cluster analysis, with the utilisation of Induced Squared Correlation Thresholding with Dimension Reduction (ISCT), which takes into consideration the squared correlation of a term within its own cluster and the nearest cluster. Our proposed method not only helps to reduce the dimension in the text data but also provides an estimate of the optimal number of clusters for a given document large enough to satisfy the ISCT conditions.