Twitter data can provide a glimpse into public opinion, but a distorted one. This is the main finding of a recent study we conducted that compares Twitter data with polling data and election results.

In times when response rates in survey research decline, and those who do participate increasingly differ from those who do not, journalists, policy makers, and academics alike have been looking into other ways to observe and measure public opinion. Twitter is quite useful for this purpose, because most users choose to keep their accounts public. This means that not only can we observe current opinion trends on Twitter, but researchers could also use it to learn how these came about. On Twitter, information is available on how different issue positions are argued, and by whom. In that sense Twitter finally allows us to observe the deliberative process that could potentially constitute the public sphere in the Habermasian sense. Then again, extant research shows that Twitter users are anything but a good representation of the public. They are usually better educated, politically interested, extravert, and male.

The case study: A referendum in the Netherlands

Yet, even if the users of the platform are not representative of the general population, their arguments and positions on various issues might still be. Maybe the smaller, elite discussion happening on Twitter is a mirror of the larger societal discussion, maybe the discussions happening on the social media platform are shaping the wider discussion itself. To gain a better understanding of the differences between the discussion on Twitter and in the wider population, we made use of a unique situation in the Netherlands in 2016. On April 6th 2016 the first ever referendum took place in the Netherlands. The population was asked whether a trade association agreement between the EU and Ukraine should be approved. Without going into too much detail, while the actual issue might seem remote to Dutch citizens, the referendum was politicized as an anti-EU referendum by a coalition of populist forces. What is important here is that a trade deal with Ukraine is an issue on which most Dutch citizens still needed to form an opinion during the referendum campaign.

Three measurements of public opinion about the same issue

For us it meant that we could study a combination of three data sources to learn more about the formation of public opinion: Traditional polling data, Twitter data, and the election result, the last of these considered the ultimate expression of the public will. The public opinion data was collected as part of the LISS panel, which is amongst the most reliable public opinion panels in the Netherlands. 1222 respondents completed the questionnaire, and according to census data, the sample consists of slightly older and higher educated people than the average Dutch person. Twitter data was collected through the API tracing all relevant hashtags. In total, we collected 30 828 tweets. Finally, we could compare both measurements of public opinion, that is Twiter and polling data, with the actual election results. While voters are not representative of the entire population either, especially in this particular referendum where the turnout was as low as 32 percent, the outcome of the election is still regarded as the will of the people in the political system.

Results

Our findings were somewhat surprising. We found that neither the polling nor Twitter data could correctly predict the outcome of the election, although both were close (see Figure 1).


Figure 1: Agreement with the treaty according to three different data sources (election result, Twitter data, and poll data) in percent

Looking at the issues that were discussed in the context of the referendum, we find a considerable overlap between Twitter data and polling data – but only if we exclude retweets (see figure 2). Once retweets are considered, the data paint an entirely different picture – it seems like the concerns of the Ukrainian people and corruption were the two most important issues that set the debate, while they played only a minor role in the polling data. In other words: we find that while Twitter at large reflects the issues in the heads of those that fill in the polls, the strong and amplified voices on Twitter most certainly do not. Hence, if you want to learn about public opinion, it is smart to ignore the highly retweeted Tweets that pop up first, and dig deeper.


Figure 2: Distribution of the most important arguments discussed in the context of the referendum on Twitter and in the poll in percent split up by voting intention (in favor or against the treaty)

We observed, however, that there were differences in whether the issues were presented in as a.pro- or a contra argument (see Figure 2). Take the issue of the EU as an example: while according to the poll those who thought the EU was important in the context of this referendum mostly voted against the treaty, on Twitter the EU was discussed in a much more positive way.

So, is Twitter a mirror of public opinion and place to study the public sphere? Yes, but it is a distorted mirror. We can take a look at it, but to be sure if what we see is true, we will have to keep looking further than that.

Judith Möller
Assistant Professor

Judith Möller is an Assistant Professor at the Amsterdam School of Communication Research. In her research, she focuses on the effects of political communication, in particular social media and digital media.

Marijn van Klingeren
Assistant Professor

Marijn van Klingeren is an Assistant Professor in Sociology at the Radboud University in Nijmegen. Her research interests include media patterns and media influences on public opinion, interethnic relations, migrant perceptions, and (Muslim) migrant integration processes.

Damian Trilling
Associate Professor

Damian Trilling is an Associate Professor at the Department of Communication Science at the University of Amsterdam and affiliated with the Amsterdam School of Communication Research. Most of his research is related to the question of how the changing media landscape affects the way in which citizens keep up with news and political information.