An Entropy-based Evaluation for Sentiment Analysis of Stock Market Prices using Twitter Data

Stock markets prediction is considered a considerably demanding task due to its notable returns as well as due to the high randomness within the stock market. Moreover, stock price alternations are primarily related to the capital circumstances and hot occasions/events. Nowadays, researchers have sufficiently improved prediction accuracy by taking into consideration news and social media. However, the existing strategies do not employ the different impacts that events may pose. Streaming data proves to be a perpetual real-time source of data analysis as information from different web sources can be carried. In this paper, we explore whether estimations, in terms of sentiment analysis derived from Twitter posts, can be correlated to the stock market prices. Initially, the daily Twitter posts are analyzed and different n-grams along with two strategies that are utilized to increase the accuracy of the classification, are applied. Spark streaming has been employed for the processing of Twitter data, while Apache Flume has been utilized for the analysis.