Can we observe public opinion on Twitter?
Twitter data can provide a glimpse into public opinion, but a distorted one. This is the main finding of a recent study we conducted that compares Twitter data with polling data and election results.
In times when response rates in survey research decline, and those who do participate increasingly differ from those who do not, journalists, policy makers, and academics alike have been looking into other ways to observe and measure public opinion. Twitter is quite useful for this purpose, because most users choose to keep their accounts public. This means that not only can we observe current opinion trends on Twitter, but researchers could also use it to learn how these came about. On Twitter, information is available on how different issue positions are argued, and by whom. In that sense Twitter finally allows us to observe the deliberative process that could potentially constitute the public sphere in the Habermasian sense. Then again, extant research shows that Twitter users are anything but a good representation of the public. They are usually better educated, politically interested, extravert, and male.
The case study: A referendum in the Netherlands
Yet, even if the users of the platform are not representative of the general population, their arguments and positions on various issues might still be. Maybe the smaller, elite discussion happening on Twitter is a mirror of the larger societal discussion, maybe the discussions happening on the social media platform are shaping the wider discussion itself. To gain a better understanding of the differences between the discussion on Twitter and in the wider population, we made use of a unique situation in the Netherlands in 2016. On April 6th 2016 the first ever referendum took place in the Netherlands. The population was asked whether a trade association agreement between the EU and Ukraine should be approved. Without going into too much detail, while the actual issue might seem remote to Dutch citizens, the referendum was politicized as an anti-EU referendum by a coalition of populist forces. What is important here is that a trade deal with Ukraine is an issue on which most Dutch citizens still needed to form an opinion during the referendum campaign.
Three measurements of public opinion about the same issue
For us it meant that we could study a combination of three data sources to learn more about the formation of public opinion: Traditional polling data, Twitter data, and the election result, the last of these considered the ultimate expression of the public will. The public opinion data was collected as part of the LISS panel, which is amongst the most reliable public opinion panels in the Netherlands. 1222 respondents completed the questionnaire, and according to census data, the sample consists of slightly older and higher educated people than the average Dutch person. Twitter data was collected through the API tracing all relevant hashtags. In total, we collected 30 828 tweets. Finally, we could compare both measurements of public opinion, that is Twiter and polling data, with the actual election results. While voters are not representative of the entire population either, especially in this particular referendum where the turnout was as low as 32 percent, the outcome of the election is still regarded as the will of the people in the political system.
Our findings were somewhat surprising. We found that neither the polling nor Twitter data could correctly predict the outcome of the election, although both were close (see Figure 1).
Looking at the issues that were discussed in the context of the referendum, we find a considerable overlap between Twitter data and polling data – but only if we exclude retweets (see figure 2). Once retweets are considered, the data paint an entirely different picture – it seems like the concerns of the Ukrainian people and corruption were the two most important issues that set the debate, while they played only a minor role in the polling data. In other words: we find that while Twitter at large reflects the issues in the heads of those that fill in the polls, the strong and amplified voices on Twitter most certainly do not. Hence, if you want to learn about public opinion, it is smart to ignore the highly retweeted Tweets that pop up first, and dig deeper.
We observed, however, that there were differences in whether the issues were presented in as a.pro- or a contra argument (see Figure 2). Take the issue of the EU as an example: while according to the poll those who thought the EU was important in the context of this referendum mostly voted against the treaty, on Twitter the EU was discussed in a much more positive way.
So, is Twitter a mirror of public opinion and place to study the public sphere? Yes, but it is a distorted mirror. We can take a look at it, but to be sure if what we see is true, we will have to keep looking further than that.