LDA Based model for mining textual features from financial news articles

The exorbitant amount of textual information circulated every day in the form of news articles, blog posts, etc., is used by humans to predict future trends. However, using this textual information to predict future drifts (crash/boom) in stocks or stock market indices in a realistic manner is very difficult through the pure use of algorithms. Also, it is quite difficult to make predictions about stock prices or market trends purely based on historical data. This paper proposes a variation of Latent Dirichlet Allocation (LDA) on a dataset of financial news items and historical data of the S&P 500 to effectively mine out words that can later be used as important features for various machine learning models with an aim to predict market trends and more specifically crash/boom in the stock market. It can be used as a topic modelling tool. In the end, a table with the top words per topic generated is illustrated which gives an insight into the result of the implementation of the proposed variation.