Learning Political Polarization on Social Media Using Neural Networks

Social media analysis is a fast growing research area aimed at extracting useful information from social media platforms. This paper presents a methodology, called IOM-NN (Iterative Opinion Mining using Neural Networks), for discovering the polarization of social media users during election campaigns characterized by the competition of political factions. The methodology uses an automatic incremental procedure based on feed-forward neural networks for analyzing the posts published by social media users. Starting from a limited set of classification rules, created from a small subset of hashtags that are notoriously in favor of specific factions, the methodology iteratively generates new classification rules. Such rules are then used to determine the polarization of people towards a faction. The methodology has been assessed on two case studies that analyze the polarization of a large number of Twitter users during the 2018 Italian general election and 2016 US presidential election. The achieved results are very close to the real ones and more accurate than the average of the opinion polls, revealing the high accuracy and effectiveness of the proposed approach. Moreover, our approach has been compared to the most relevant techniques used in the literature (sentiment analysis with NLP, adaptive sentiment analysis, emoji- and hashtag- based polarization) by achieving the best accuracy in estimating the polarization of social media users.

SECTION I. Introduction

Every day millions of people use social media and produce huge amount of digital data that can be effectively exploited to extract valuable information concerning human dynamics and behaviors. Such data, commonly referred as Big Data, contains valuable information about user activities, interests, and behaviors, which makes it intrinsically suited to a very large set of applications [1]. Big Social Data analysis is a subfield of Big Data analysis aimed at studying the interactions of users on social media for extracting useful information, such as moods or opinions on topics or events of interest [2].

This paper presents a new methodology, namely IOM-NN (Iterative Opinion Mining using Neural Networks), for estimating the polarization of public opinion on political events characterized by the competition of factions or parties. It can considered as an alternative technique to traditional opinion polls, since it is able to capture the opinion of a larger number of people more quickly and at a lower cost. In particular, IOM-NN uses an automatic incremental procedure based on feed-forward neural networks for analyzing the posts published by social media users. Starting from a limited set of classification rules, created from a small subset of hashtags that are notoriously in favor of specific factions, our methodology iteratively generates new classification rules. A classification rule allows to determine if a post is in favor of a faction based on the words/hashtags it contains. Then, such rules are used to determine the polarization of social media users - who wrote posts about the political event - towards a faction.

The proposed methodology has been applied to two case studies for analyzing the polarization of a large number of Twitter users during the 2018 Italian general election and the 2016 US presidential election. The results obtained by IOM-NN have been compared to opinion polls collected before voting and the most relevant techniques used in the literature (i.e., sentiment analysis with NLP [3], adaptive sentiment analysis [4], emoji-based polarization [5], hashtag-based polarization [6]). The results achieved by IOM-NN are very close to the real ones and more accurate than opinion polls and other relevant techniques, revealing the high accuracy and effectiveness of the proposed approach. For example, considering the 2018 Italian general election and the four parties that received the highest number of votes (M5S, PD, LEGA, FI), IOM-NN achieved a mean average error (MAE) of 1.13 percentage points and a log accuracy ratio (LogAcc) very close to 1. Opinion polls achieved a MAE of 3.74 percentage points and a LogAcc of 0.81. Compared with the other existing techniques, IOM-NN turned out to be the most accurate in forecasting the winning candidate. For example, considering the 2016 US presidential election, IOM-NN has been able to correctly identify the winning candidate in 8 out of 10 states, while the other techniques identified the winner in up to 6 out of 10 states.

Compared to existing techniques, our methodology includes the following innovative aspects: $i$ ) it allows to classify a much high number of users, which in our case studies results to be ten times larger than that involved in traditional opinion polls; $ii$ ) it measures the political consensus of a faction considering the number of users supporting it, without being influenced by users who published a large volume of posts; $iii$ ) it is able to process posts written in any language without the need to use dictionaries or translation systems; $iv$ ) at the best of our knowledge, it is the first methodology in this research field that exploits an iterative learning approach to increase the amount of classified data; $v$ ) collected data has been statistically validated for assessing the representativeness of users involved in the analysis.

This manuscript significantly extends a previous work [7] in the following main aspects: $i$ ) it provides an in-depth definition of the steps of the methodology and a formal description through pseudo-code of the algorithms used for classifying the posts and predicting the users’ polarization (see Sections III-B and III-C); $ii$ ) it presents and discusses a new case study on which the methodology has been applied and tested (see Section IV); $iii$ ) it includes the evaluation of the statistical significance of the collected data for the two case studies; and $iv$ ) it reports more extensive and detailed tests and comparisons with relevant techniques used in the literature (see Section IV).

The remainder of the paper is organized as follows. Section II discusses related work. Section III describes the proposed methodology. Section IV presents the case studies and Section V concludes the paper.

SECTION II. Related Work

In recent years, social media analysis is arousing great interest in various scientific fields, such as sociology, political science, linguistics, and computer science [8]. In this section, we focus on the main techniques and algorithms proposed for measuring public opinion and predicting election results through social media. As suggested in [9] and [10], the existing techniques can be divided into three main categories: volume-based, sentiment-based and network-based. For each category, the main proposed solutions and their differences with respect to our technique are discussed.

Volume-based techniques counts the number of mentions (e.g., posts, likes, retweets) related to a candidate/party for predicting the election results. In many cases, such techniques analyze social media data for predicting the outcome of an election. For example, Gaurav et al. [11] proposed a technique based on moving average aggregate probability, which infers the results of an election by counting how many times a candidate’s name is mentioned in tweets. Tumasjan et al. [12] used micro-blogging data for understanding how people express their political orientation, showing a high closeness between the volume of tweets mentioning a party and the election results. Burnap et al. [13] analyzed the volume of mentions for calculating an overall score for each party.

Unlike volume-based techniques, which consider the number of posts in favor of a faction, our technique takes into account the number of users supporting a faction. In this way, IOM-NN obtains more accurate results since it is not influenced by users who published a large number of posts.

Sentiment- or opinion-based techniques exploit natural language processing (NLP) or text mining algorithms for understanding the opinion of users towards political candidates or parties. Such techniques result to be more advanced than the volume-based ones as they analyze the textual content of posts to calculate a score.

The techniques based on natural language processing consider the hierarchical structure of a text to understand its meaning and sentiment. For example, Oikonomou and Tjortjis [3] exploited Textblob,¹ a Python library for natural language processing, to predict the outcome of USA presidential elections in three states of interest (i.e., Florida, Ohio and North Carolina). Wong et al. [14] combined convex optimization techniques with SentiStrength,² a lexicon-based sentiment analysis tool, for modeling the political behaviors of users by analyzing tweets and retweets. Alashri et al. [15] analyzed Facebook posts about the 2016 US presidential election with CoreNLP³ [16], one of the most popular tool for natural language processing, to calculate a score for each political candidate.

Techniques based on text mining discover the sentiment of a text by considering only the words it contains without analyzing its structure. For example, El Alaoui et al. [4] proposed an adaptive sentiment analysis approach that generates dictionaries from tweets classified as positive/negative for the different factions. Such dictionaries are then used to calculate a score for each faction. Similarly, Marozzo and Bessi [6] calculated a polarization score for each faction by considering only the hashtags in tweets labeled as positive. Chin et al. [5] exploited the emojis contained in a post to determine its sentiment (e.g., positive or negative). Other studies exploited machine learning techniques for discovering the political orientation of users, such as classification models based on the Naïve Bayes algorithm [17] or logistic regression [18].

IOM-NN is a text mining technique that uses bag-of-words and neural networks to classify posts, and consequently users who wrote such posts about a political event. Compared to existing text mining techniques, its iterative approach allows to greatly increase the number of classified posts, while the use of neural networks permit to automatically discover classification rules with a high level of accuracy. With regard to NLP techniques, it is worth noting that their usability and accuracy depend on the specific tool used and the supported languages. In fact, the most popular tools for natural language processing (e.g., CoreNLP) support sentiment analysis only for English texts. Instead IOM-NN classifies users by analyzing the words/hashtags contained in the posts that can be written in any language, without using dictionaries or translation systems.

Network-based techniques analyze the network structure of social media users, which support or discuss about certain candidates or parties, for understanding the dynamics of public opinion. Such analysis can provide useful insights for estimating the standing of political events or identifying the opinion leaders on a social media platform [19]. In fact, some studies have demonstrated a relation between the centrality of political candidates on social networks and their electoral consensus [20], [21]. However, it should be noticed that such techniques require the use of specific data that represents the social network structure, which is often visualized through graphs or sociograms. In our study, we collected and analyzed tweets containing specific keywords or hashtags on the political event under analysis, without capturing the structure of the related social network. For this reason, in this paper we cannot make a comparison with network-based techniques.

Some studies highlighted the issues related to the use of social media data for predicting the outcome of political events, which are language barrier, misclassification, data imbalance and reliability [10]. During the design of IOM-NN we faced such issues by proposing the following solutions:

• Language barrier. Our technique classifies users by analyzing the words/hashtags contained in their posts, regardless of the language used to write them.

• Misclassification. Starting from a limited set of classification rules, created from a small set of hashtags that are notoriously in favor of specific factions, the methodology iteratively generates new classification rules.

• Data imbalance. To avoid the learning process being biased towards majority classes, a random under-sampling approach is used to balance the dataset at each training phase (see Algorithm 1).

• Data reliability. The statistical significance of the collected data has been evaluated for assessing the representativeness of users, i.e., understanding whether they can be considered voters in the political event under analysis.

Algorithm 1

Classification of the Posts

SECTION III. Proposed Methodology (IOM-NN)

As mentioned in Section I, IOM-NN is a methodology for estimating the polarization of public opinion during a political event, which is characterized by the rivalry of different factions. As shown in Figure 1, the proposed methodology consists of three main steps:

1. Collection of posts: posts are collected by using a set of keywords related to the selected political event (see Section III-A).

2. Classification of posts: the collected posts are then classified by using an incremental procedure implemented through neural networks (see Section III-B).

3. Polarization of users: the classified posts are analyzed for determining the polarization of users towards a faction (see Section III-C).

FIGURE 1.

Execution flow of IOM-NN.

For each step, a formal description and practical examples are provided in the following sections. For the sake of clarity, Table 1 reports the meaning of the main symbols used to describe the different steps.

TABLE 1 Meaning of the Most Important Symbols Used in the Proposed Methodology

A. Collection of Posts

A political event $\mathcal {E}$ is characterized by the rivalry of different factions $F=\{f_{1}, f_{2}, \ldots, f_{n}\}$ . Examples of political events and relative factions are: $i$ ) municipal election, in which a faction supports a mayor candidate; $ii$ ) parliament election, in which a faction supports a party; $iii$ ) presidential election, in which a faction supports a presidential candidate. The posts are collected by using the keywords that people commonly use to refer the political event $\mathcal {E}$ on social media. Such keywords $K$ can be divided in two groups:

• $K_{context}$ , which contains generic keywords that can be associated to $\mathcal {E}$ without referring to any specific faction in $F$ .

• $K^\oplus _{F} = K^\oplus _{f1} \cup \ldots \cup K^\oplus _{fn}$ , where $K^\oplus _{fi}$ contains the keywords used for supporting $f_{i} \in F$ (positive faction keywords).

The keywords in $K$ are given as input to public APIs provided by social media platforms, which permit to collect posts containing one or more keywords. Posts are not collected in real time, but downloaded at a given time after their publication (e.g., 24 hours). In this way, we are able to get some statistics related to the popularity of a post (e.g., number of shares, number of likes). Since data collection is usually a continuous process, new keywords can be discovered and integrated in $K$ during the collection procedure. It is important to highlight that obtaining a representative collection of posts depends on two factors: $i$ ) the quality and the number of keywords used; $ii$ ) the amount of data that can be downloaded from social media. Regarding the latter factor, it is increasingly difficult to obtain complete data from social media platforms due to the restrictions introduced for protecting the privacy of users.

The collected posts are pre-processed before the analysis. In particular, they are modified and filtered as follows:

• The text of posts is normalized by transforming it to lowercase and replacing accented characters with regular ones (e.g., IOVOTOSI or iovotosí $\rightarrow$ iovotosi).

• Words are stemmed for allowing matches with declined forms (e.g., vote or votes or voted $\rightarrow$ vot).

• Stop words are removed from text by using preset lists.

• All the posts written in a language different from the one(s) spoken in the nation(s) hosting the considered political event are filtered out.

The output of this step is a collection of posts $P$ . Figure 2 shows an example of how posts are collected using keywords about the 2016 US presidential election. Some of these keywords are generic (e.g., election2016), and others are used to support a specific candidate (e.g., #imwithher for Clinton and #votetrump for Trump).

FIGURE 2.

Example of how the collection of posts step works.

Before the analysis, the statistical significance of the collected data has to be evaluated. We studied the age, gender and geographical distribution of social media users who generated such data. The aim is to assess the users’ representativeness by understanding whether they can be considered voters of the political event under analysis (more details in Sections IV-A.1 and IV-B).

B. Classification of Posts

Algorithm 1 shows the pseudo-code used for classifying the posts. The input is composed of: the posts $P$ generated in the previous step, the set of positive faction keywords $K^\oplus _{F}$ , the maximum number of iterations $max_{iters}$ , the minimum increment of the classified posts $eps$ at each iteration, and a threshold $th$ . The output is a collection of posts $C$ that have been classified in favor of a faction.

The algorithm is divided in two parts. The fist part (lines 1-9) performs the preliminary iteration (iteration 0). At this iteration, IOM-NN exploits the set of positive faction keywords ($K^\oplus _{F}$ ) for classifying a part of the posts. Specifically, it classifies a post in favor of a faction if it contains only positive keywords for such faction. In general, at the end of this iteration, a small amount of posts are classified, since not all users use keywords in $K^\oplus _{F}$ for declaring their support to factions. The second part (lines 10-21) iteratively generates new classification rules for classifying other posts. At each iteration, such rules are inferred by exploiting the posts that have been classified at the previous iterations. In the following of this section, we discuss in detail the code of Algorithm 1.

The algorithm initializes an empty set $C^{0}$ for storing the classified posts and builds a classification model $M^{0}$ based on the positive faction keywords $K^\oplus _{F}$ (lines 1-2). The model defines a set of rules for calculating a binary vector $v_{b}$ , where $v_{b}[i]$ is 1 if $p$ contains at least one keyword from $K^\oplus _{fi}$ , 0 otherwise. The algorithm iterates (lines 3-7) on each post $p$ in $P$ performing the following operations:

• classifies $p$ using $M^{0}$ , which produces a vector $v_{b}$ (line 4);

• if $p$ is in favor of a single faction $f$ (lines 5-6), the classified post (i.e., a pair $\langle p, f\rangle$ ) is added to $C^{0}$ (line 7).

At the end of iteration 0, the set of classified posts $C^{0}$ is stored in $C$ (line 8). The set of unclassified posts ($N^{0}$ ) is obtained as the difference between $P$ and $C^{0}$ (lines 9). It is important to note that the number of keywords has to be balanced among the different factions for avoiding a classification process biased towards a faction.

The second part of the algorithm (lines 10-21) performs at most $max_{iters}$ iterations. Specifically, at the $i$ -th iteration, the following operations are performed:

• It initializes an empty set $C^{i}$ for storing the classified posts at $i$ -th iteration (line 11).

• It builds a classification model $M^{i}$ by training a neural network using the classified posts at previous iterations $C^{0} \cup \ldots \cup C^{i-1}$ (lines 12). The training set is balanced by using a random under-sampling approach to avoid a learning process biased towards majority classes.

• For each unclassified post at the previous iteration $N^{i-1}$ (line 13), the algorithm classifies $p$ using $M^{i}$ , which produces a vector of probabilities $v_{p}$ (line 14), where $v_{p}[i]$ is the probability that $p$ supports $f_{i}$ . If the maximum value of $v_{p}$ is greater than the given threshold $th$ , the post is assigned to the most likely faction $f$ (lines 15-16) and added to $C^{i}$ (line 17).

• The set of classified posts $C^{i}$ is added to $C$ (line 18), and the unclassified posts $N^{i}$ are obtained as difference between $N^{i-1}$ and $C^{i}$ (line 19).

• If the ratio between the size of $C^{i}$ and the size of $N^{i-1}$ is lower than $eps$ or greater that $1-eps$ , then it breaks the loop (lines 20-21).

Finally, the algorithm returns the dictionary $C$ containing all the posts classified at the various iterations (line 22). Since the parameters of the neural network are randomly initialized, IOM-NN repeats the post classification phase with a new random seed for $n_{seeds}$ times, in order to reduce the risk of getting stuck in local minima or saddle points.

Figure 3 shows how the post classification algorithm (Algorithm 1) works starting from a set of posts $P$ . At the iteration 0, the classification model $M^{0}$ is created using the faction keywords $K_{F}$ . This model is used to classify $P$ , which generates two subsets for classified ($C^{0}$ ) and unclassified posts ($N^{0}$ ) respectively. At iteration 1, a new model $M^{1}$ is trained using $C^{0}$ and is used to classify the unclassified posts generated at the previous iteration ($N^{0}$ ). The classification process splits $N^{0}$ in two new subsets: $C^{1}$ for classified posts and $N^{1}$ for unclassified ones. Then, at the $i-th$ iteration, the model $M^{i}$ is trained using $C^{0} \cup \ldots \cup C^{i-1}$ . For example, at iteration 2 the model $M^{2}$ is trained using $C^{0} \cup C^{1}$ . The process is repeated in subsequent iterations until the ratio between the size of $C^{i}$ and the size of $N^{i-1}$ is lower than $eps$ or greater that $1-eps$ . At the end, the whole set of classified posts is obtained as the union of the $C^{i}$ produced at each iteration, while the remaining posts ($N^{2}$ ) are classified as neutrals.

FIGURE 3.

Example of the classification of posts algorithm terminating in three iterations.

Table 2 shows an example of post classification on ten tweets about the 2016 US presidential election. The input of the algorithm is composed of a set of tweets regarding the political event and a set of faction keywords $K^\oplus _{F}$ :

• $K^\oplus _{Clinton} =$ {#voteHillary, #imwithher, #strongertogether, #hillary2016}

• $K^\oplus _{Trump} =$ {#voteTrump, #maga, #americafirst, #wakeupamerica}

TABLE 2 Example of How the Classification of Posts Algorithm Works

At iteration 0, $K^\oplus _{F}$ is used to generate $M^{0}$ , which allows to classify 5 tweets. At iteration 1, classified tweets at iteration 0 are used to train $M^{1}$ . This model generates new classification rules, such as:

• since Donald Trump has been accused of sexual assault by some women, tweets with keywords #sex and #woman are classified in favor of Clinton;

• similarly, since Hillary Clinton contravenes the federal laws by using personal email account for government business, tweets with keywords email and #hillary are classified in favor of Trump.

At iteration 2, the algorithm learns other classification rules about immigration, a topic on which the two candidates had an opposite opinion. The iterative learning process ends when the algorithm is no longer able to generate new classification rules and therefore to classify new tweets.

C. Polarization of Users

Algorithm 2 shows the pseudo-code of the algorithm used for determining the polarization of users. The input is a collection of classified posts $C$ (i.e., output of Algorithm 1), a filtering function $filter$ and its parameters $par_{f}$ , and a polarization function $polarize$ and its parameters $par_{p}$ . The output is composed of a collection of classified users $U$ and a faction score ($S$ ) containing the polarization percentages of each faction.

Algorithm 2

Prediction of User Polarization

As first step, the classified posts are aggregated by user to produce a dictionary ($C_{U}$ ), which contains the list of classified posts $P_{u}$ for each user $u$ (line 1). Two empty variables are initialized for storing the output (lines 2-3).

On each pair $\langle u, P_{u} \rangle$ of $C_{U}$ , the algorithm performs the following operations (lines 4-7):

• It filters out all the pairs that do not match the criteria defined by the $filter$ function (line 5). For example, users who published a number of posts below a given threshold are skipped.

• Using the classified posts $P_{u}$ , it computes $v^{u}_{s}$ a vector containing the score of user $u$ for each faction (line 6). The score vector is calculated by using the function $polarize$ .

• It adds the pair $\langle u, v_{s}\rangle$ to $U$ (line 7).

Then, the algorithm calculates the overall faction score $S$ as the normalized sum of the user vector scores $\langle u, v^{u}_{s} \rangle$ (lines 8-10). Finally, the two output are returned (line 11).

The filter and polarize functions, used for analyzing the data collected for our case studies (see Section IV), have been configured as follows. Specifically, a user $u$ is considered only if he/she fulfills the following criteria: $i)~u$ posted at least minPosts on the political event of interest; $ii$ ) it exists a faction $f$ for which $u$ has published more than 2/3 of his/her posts. For each user $u$ , the $polarize$ function returns a vector score as follows: the percentage of posts written by $u$ in favor of preferred faction $f$ , 0 for the other factions.

Figure 4 shows how the user polarization algorithm (Algorithm 2) works on the classified posts shown in Table 2. For each user, the posts if favor of Clinton and Trump are counted. Users who fulfill the criteria of filter function are considered and added to the set of classified users $U$ . Then $U$ is combined and normalized to obtain the vector $S$ containing the overall polarization percentages.

FIGURE 4.

Example of how the user polarization algorithm works.

SECTION IV. Case Studies

In this section we describe and analyze two case studies: the 2018 Italian general election and 2016 US presidential election. In both case studies, for each faction $f_{i}$ we defined three set of keywords $K^\oplus _{fi}$ , $K^\circleddash _{fi}$ and $K_{fi}^{\bigcirc\!\!\!{\circ}}$ that are respectively positive, negative and neutral keywords for faction $f_{i}$ . For example, for the Movimento 5 Stelle ($M5S$ ) faction in the 2018 Italian general election, $K^\oplus _{M5S}$ contains keywords used to clearly support $M5S$ party (e.g., #iovotoM5S), $K^\circleddash _{M5S}$ contains keywords to speak negatively about $M5S$ (e.g., #maiM5S), $K_{M5S}^{\bigcirc\!\!\!{\circ}}$ contains neutral keywords for $M5S$ (e.g., m5s or movimento5stelle).

As described in Section III, IOM-NN exploits only positive faction keywords ($K^\oplus _{f_{i}}$ ) for classifying posts and then for determining the polarization of users. For evaluating the accuracy of IOM-NN, we carried out an extensive comparison with the most relevant techniques used in literature:

1. Sentiment analysis with NLP [3], [15]. For each post, we used CoreNLP [16] for calculating a sentiment score that ranges from 0 (very negative) to 4 (very positive). The neutral keywords ($K_{fi}^{\bigcirc\!\!\!{\circ}}$ ) are then used for grouping posts and calculating an overall score for each faction.

2. Adaptive sentiment analysis [4]. Starting from the positive and negative keywords of each faction ($K^\oplus _{fi}$ and $K^\circleddash _{fi}$ ), this technique generates two word-polarity dictionaries, which are built from a set of posts containing such positive and negative keywords. Also in this case, a score for each faction is returned.

3. Emoji-based polarization [5]. This technique groups the posts of each faction by using keywords ($K_{fi}^{\bigcirc\!\!\!{\circ}}$ ), then classifies their sentiment by using emojis and returns a score for each faction.

4. Hashtag-based polarization [6]. The posts are classified as in favor of a given faction based on the positive faction keywords ($K^\oplus _{fi}$ ). Then the posts are aggregated by users and the polarization of each user is computed.

To allow a direct comparison with the real percentages, the results obtained by the different techniques have been normalized with respect to the sum of the real ones.

A. 2018 Italian General Election

Here we discuss the case study carried out to analyze the polarization of a large number of Twitter users during the 2018 Italian general election. Twitter users have been classified using the polarization rules extracted from our methodology and the results have been compared to: $i$ ) official results; $ii$ ) main opinion polls collected before voting; and $iii$ ) other techniques present in the literature.

Italians voted to elect 630 deputies and 315 senators of the XVIII legislature: the results decreed the center-right coalition as the most voted, with about 37% of votes, while the most voted list was the Movimento 5 Stelle, which received over 32% of votes. The electorate was composed of 50,782,650 voters for the Chamber of Deputies and 46,663,202 for the Senate,⁴ with a turnout of about 73%, the lowest in Italian republican history.

In order to assess the validity of the proposed methodology, the analysis we carried out focused on the four most successful political factions, in decreasing order of consensus: M5S (Movimento 5 Stelle), PD (Partito Democratico), LEGA, FI (Forza Italia). In the following, we show how the classification model has been trained and discuss the main achieved results.

1) Models Training and Iteration-Level Results

IOM-NN has been used to classify 60,782 tweets posted by 21,883 users from February 1, 2018 to March 3, 2018 (the day before the election). By following the approach described in [6] we can assess that collected data is statistically significant for the event under analysis, since:

• All the tweets under analysis have been written in Italian, that means they have the lang field set to it (Italian). With very few exceptions, the Italian language is used only by Italians people who reside in Italy or abroad.⁵

• 92% of the users who set a location in their profile, specified a region in Italy. Moreover, there is a strong correlation between the number of users that can be assigned to a region and the number of people actually living in that region according to official statistics (the Pearson correlation coefficient is equal to 0.8 with a confidence interval of 99%).

• About 98% of the Italian social media users are adults and equally divided by gender (51.2 females and 48.8 males)⁶.

IOM-NN exploits the following positive faction keywords for analyzing the collected data:

• $K^\oplus _{M5S} =$ {#iovotom5s,#m5salgoverno, #dimaiopresidente}

• $K^\oplus _{PD} =$ {#sceglipd,#iovotopd,#pdvinci}

• $K^\oplus _{LEGA} =$ {#4marzovotolega,#iovotolega, #salvini- premier}

• $K^\oplus _{FI} =$ {#berlusconipresidente,#votoforzaitalia, #4marzovotoforzaitalia}

The threshold $th$ and the minimum increment $eps$ have been set to 0.9 and 5% respectively. In our test, the post classification algorithm terminated in 4 iterations by annotating 23,997 tweets, which represents about 39.5% of the total. Table 3 shows the obtained results at each iteration by specifying the number of classified and unclassified tweets, the ratio $\frac {|C^{i}|}{|N^{i-1}|}$ and the accuracy of the neural network.

TABLE 3 Partial Results for Each Iteration Achieved by IOM-NN (2018 Italian General Election)

2) Polarization of Users and Final Results

The algorithm described in Section III-C has been used for analyzing the users who have written the 23,997 classified tweets so as to determine their polarization degree towards the considered factions. The first three rows of Table 4 shows a comparison between the official results, the average of the latest polls and the percentages obtained by IOM-NN. We evaluated the accuracy through different statistical indexes, comparing the obtained results with the latest opinion polls published before the elections. Considering the four most supported parties, our methodology obtained the following approval percentages: M5S 31.64%, PD 19.89%, LEGA 18.45%, and FI 12.80%. These results are extremely close to the real ones (i.e., M5S 32.68%, PD 18.72%, LEGA 17.37%, FI 14.01%), even more than the average of polls. In addition, the obtained results are characterized by very good values of log accuracy ratio, as well as a negligible value of mean percentage and absolute errors. In particular, our methodology achieved a mean average error (MAE) of 1.13 percentage points and a log accuracy ratio (LogAcc) very close to 1. On the other hand, opinion polls achieved a MAE of 3.74 percentage points and a LogAcc of 0.81, which confirm the ability of the proposed methodology to forecast election results. Figure 5 shows an info-graphic about the comparison of the real percentages, opinions polls and obtained results.

TABLE 4 Obtained Percentages and Accuracy Evaluation on the 2018 Italian General Election Dataset

FIGURE 5.

Comparison among real percentages, opinions polls and IOM-NN results (2018 Italian general election).

Table 4 also presents the results obtained by the other techniques in the literature (rows 4-7). These techniques have been configured with the positive faction hashtags used by IOM-NN (see $K^\oplus _{M5S}$ , $K^\oplus _{PD}$ , $K^\oplus _{LEGA}$ and $K^\oplus _{FI}$ ) and the following negative and neutral faction keywords:

• $K^\circleddash _{M5S} =$ {#nom5stelle, #rimborsopolim5s, #maim5s} and $K_{M5S}^{\bigcirc\!\!\!{\circ}} =$ {m5s, movimento5stelle, dimaio}

• $K^\circleddash _{PD} =$ {#nonvotopd, #maipd, #bastapd} and $K _{PD}^{\bigcirc\!\!\!{\circ}} =$ {pd, partitodemocratico, renzi}

• $K^\circleddash _{LEGA} =$ {#maiconsalvini, #iononvotolega} and $K _{LEGA}^{\bigcirc\!\!\!{\circ}} =$ {lega, salvini, leganord}

• $K^\circleddash _{FI} =$ {#maipiuberlusconi, #stopberlusconi} and $K_{FI}^{\bigcirc\!\!\!{\circ}} =$ {forzaitalia, berlusconi}

Compared to such techniques, IOM-NN turned out to be the most accurate in estimating the voting percentages, outperforming the competitors in terms of achieved LogAcc, MAPE and MAE. Compared to the emoji- and hashtag-based techniques, IOM-NN is able to classify a much greater number of tweets and users, which ensures greater statistical representativeness of data and robustness of results.

It should be noticed that, differently from other techniques (i.e., sentiment analysis with NLP and adaptive sentiment analysis), IOM-NN is not volume-based. This means that, since it gives the same weight to each user regardless of the number of published posts, the results obtained are not influenced by users who published a large number of posts. Since CoreNLP provides a well-trained model for sentiment analysis only for the English language, all the tweets downloaded in Italian have been translated in English before being processed.

B. 2016 Us Presidential Election

After the presentation of the Italian use case, here we discuss the analysis we carried out on the 2016 US presidential election, which was characterized by the rivalry between Hillary Clinton and Donald Trump.

The analysis has been performed on data collected for ten US Swing States: Colorado, Florida, Iowa, Michigan, Ohio, New Hampshire, North Carolina, Pennsylvania, Virginia, and Wisconsin. Swing states are those characterized by greater political uncertainty, in which neither major political party holds a lock on the outcome of presidential elections. These states are considered of strategic importance, as their votes have a high probability of being the deciding factor in a presidential election. For each state, data have been collected through the standard Search Twitter API, which allows for collecting tweets published in a given area or place. Overall about 2.5 million of tweets, posted by 521,291 users, have been collected from October 10, 2016 to November 7, 2016 (the day before the election). From such data we filtered out all the tweets posted by users with a not defined location or with a location that does not belong to any of the considered states. Filtered data (818,403 tweets posted by 141,959 users) are statistically significant for the event under analysis, since:

• All the tweets under analysis have the lang field set to en (English).

• For each state, there is a strong correlation between the number of analyzed users and the number of people actually living in that state according to official statistics (the Pearson correlation coefficient is equal to 0.95 with a confidence interval of 99%).

• About 94% of the social media users in USA are adults (at least 18 years old) and almost equally divided by gender (42.7% females and 57.3% males).⁷

The following keywords have been used in our experiments:

• $K^\oplus _{Clinton} =$ {#voteHillary, #imwithher, #strongertogether, #hillary2016},

• $K^\circleddash _{Clinton} =$ {#neverhillary, #lockherup}

• $K_{Clinton}^{\bigcirc\!\!\!{\circ}} =$ {clinton, hillary, democrats, dems}

• $K^\oplus _{Trump} =$ {#voteTrump, #maga, #americafirst, #wakeupamerica}

• $K^\circleddash _{Trump} =$ {#nevertrump, #dumpfortrump}

• $K_{Trump}^{\bigcirc\!\!\!{\circ}} =$ {trump, donald, republicans, gop}

Table 5 shows the results obtained using IOM-NN in comparison with the real voting percentages, the main opinion polls, and the other related techniques. For each state in the table, we reported the results obtained by the two candidates, where “${C}$ ” stands for Clinton and “${T}$ ” for Trump. In addition, for the winning candidate that has been correctly identified, the value is written in bold. As shown in Figure 6, IOM-NN is able to correctly identify the winning candidate in 8 out of 10 cases, outperforming the opinion polls that correctly classifies 6 out 10 states.

TABLE 5 Obtained Percentages and Accuracy Evaluation on the 2016 US Presidential Election Dataset. For Each Technique, When the Winning Candidate is Correctly Identified, the Percentage is Written in Bold

FIGURE 6.

Comparison among the real winning candidate and that identified by IOM-NN and opinions polls. The Democratic Donkey symbolizes the party of Hillary Clinton, while the Republican Elephant that of Donald Trump.

Also in comparison with the other techniques, IOM-NN turned out to be the most accurate in discovering the winning candidate. In fact, emoji- and hashtag-based techniques classified a much smaller amount of tweets and correctly identified the winning candidate in 6 and 3 cases respectively. The results of the adaptive sentiment analysis were very poor, since it correctly identified the winning candidate in only one case, while sentiment analysis with NLP produced right predictions in 4 out 10 cases. Compared to IOM-NN, some techniques (i.e., adaptive sentiment analysis and opinion polls) achieved slightly better results in terms of log accuracy ratio, MAPE, and MAE, because their predictions are quite balanced by assigning almost the same score to the two candidates in the different states.

SECTION V. Conclusion

This paper proposes a methodology, named IOM-NN, for estimating the polarization of public opinion regarding political events characterized by the competition of factions or parties. The designed methodology uses an automatic incremental procedure based on feed-forward neural networks for analyzing the posts published by social media users. IOM-NN can be considered an alternative technique to traditional opinion polls since it is able to capture the opinion of a larger number of people more quickly and at a lower cost. In addition, it can capture the public opinion for topics perceived as embarrassing or offensive, for which people are reluctant to declare their true opinion during the polls.

IOM-NN has been validated through two case studies that analyzed the polarization of a large number of Twitter users during the 2018 Italian general election and 2016 US presidential election. The achieved results are very close to the real ones and more accurate than the average of the opinion polls, thus revealing the high accuracy and effectiveness of the proposed approach. Moreover, our approach has been compared to the most relevant techniques used in the literature (sentiment analysis with NLP, adaptive sentiment analysis, emoji- and hashtag- based polarization). Results show that IOM-NN achieved the best accuracy in estimating the polarization of social media users.

As future work, IOM-NN can be adapted to process real-time data, also coming from different sources (e.g, e-commerce and news sites blogs). Furthermore, its effectiveness can be evaluated also in other application domains, such as reputation evaluation of companies and competitive product analysis.