The latest French and American presidential election campaigns provided a splendid opportunity to determine the intercultural relevance of populist discourses. In France, Marine Le Pen and Jean-Luc Mélenchon, a right-wing and a left-wing outsider, competed with Emmanuel Macron, a relative newcomer with strong ties to the political establishment. This constellation was mirrored in the US, with Bernie Sanders and Donald Trump as political mavericks from left and right fringes of the political spectrum, and Hillary Clinton as the representative of the establishment. Since none of the outsiders had the support of the mainstream media, their mobilization strategies relied on communication via social media platforms. Their respective Twitter feeds quickly raised the suspicion that the discourse of Left and Right outsiders sounded populist. Could it be that the successful outsiders all used a populist narrative despite their stark ideological differences? Our findings have just been published in the European Journal of Communication.

To advance populism research using these campaigns, we needed comparable and representative samples of each candidate’s communication. Tweets are ideal in this respect because they were widely used in both campaigns, and reflected the political discourse of each candidate. They were also disseminated by other media. Better than any single speech, the tweets of a politician collected over a significant period reveal the peculiarities of their political discourse and ipso facto their ideological positions.

We downloaded the tweets of the six politicians (using the Twitter API) during a 13-month window around the respective elections (12 months prior, 1 month after). The length of the period ensured we capture the entire campaign that slowly built-up, including primaries. The analyses relied on about 25 000 collected tweets, between 3 000 and 6 000 for each candidate. The aim was to scan the tweets for features that indicate the use of the populist narrative in association with a left or right ideological touch.

Automated text analysis provides a reliable method for a rough, quantitative comparison of such a large amount of text under the condition that the constructs – right and left variants of populism – are well operationalized. Populism has two theoretical components, the elite and the people, for which linguistic realizations can exist in the tweets. The expressions that are pertinent for each must be listed and become the search terms in the automated analysis.

From construct to instrument

Operationalization is a qualitative process of analyzing a subset of the tweets of each candidate (roughly 10%) to circumscribe the phraseology with which they addressed “elite” including subcategories and “the people”. This process resulted in the creation of a lexicon (or dictionary). Questions whether to include a term or not arise constantly during this process. There are a multitude of ways to address either of the constructs in speech. Think for instance of the term rigged, which Trump often used in combination with political institutions (system, election, primaries) to express that they do not work fairly. Another example is the top 1 %, an expression often used by Sanders to epitomize the very rich as the enemy of the people in a Left version of populism.

Even core terms of the populist vocabulary are ambiguous in political parlance: in some phrases they acquire a populist meaning in others not. One way to solve this is by not only defining terms to include but also terms to exclude. An example is the French term le système (the system): In many tweets it was used as part of a compound term such as le système des retraites (the pension system) which lacks any populist meaning whereas in others contexts le système means something like “the oppressive/corrupt system of political power”. That means the frequent term le système cannot be incorporated in the lexicon without excluding several compounds at the same time.

The aim of efforts invested in the creation of a valid lexicon is to increase as much as possible precision and recall. Precision is high when the lexicon flags only relevant tweets whereas recall means that no relevant tweets are missed even though irrelevant ones are mistakenly flagged as well. There can be a trade-off between both. Precision turned out to be the bigger concern for our analysis due to the ambiguities and context-sensitivity of expressions. We decided to include a couple of terms that do not have a populist meaning in all contexts for the sake of improving recall event though this lowered precision. It remains of course a matter of scientific judgement to what extent this can be tolerated.

Cross-cultural validity

In a cross-cultural study like this one, the equivalence of lexicons across languages is furthermore crucial. That cannot be achieved by simply translating all the words selected for one language to the other. Simply put, the same meaning is expressed with partly different words and phrases from one language and culture to another. So, it is crucial to compile two lexicons independently while checking constantly if words selected in one language are also valid in the other. French and English are relatively compatible in that respect. A core set of populist terms are direct translations in the other language. But, of course, there was also a portion of specific expressions in each culture and even for each politician. We also took care to keep the lengths of the lexicons approximately equal even though the number of terms is secondary as long as the range of meanings captured by the lexicons is equivalent. Cross-cultural equivalence can only be established when the researchers have excellent knowledge of the contexts they study. Only then are they in the position to recognize the relevant terms and expression for the lexicon. As an additional measure of precaution, we expressed how often the politicians used populist terms in raw numbers and as divergence from their country mean. That way, the measure can be purged of potential culture-specific influence and measurement influence.

Concluding remarks

To end on a happy note, the efforts made in designing the instrument seemed to pay off. The comparison of our findings with those of recently published studies investigating some of the same cases (Clinton, Sanders, Trump), and using the same lexicon-based method, revealed great similarity in the classification of the politicians with respect to how often they use the different populist categories. This is a good external criterion for evaluating our instrument, and it looks like it, at least in its English version, passed the test. Beside the difficulties with precision and recall of automated tools, any worthwhile advancement in this area presupposes a consensus about the manifestations of specific ideologies in political speech. This condition, however, seems far from easy to meet in today’s polarized political climate.

Peter Maurer
Website

Dr Peter Maurer is a Research Associate at the University of Trier (Germany). He studies political communication and journalism in digital contexts. In particular, his research interests include populist communication, networks of politicians and journalists, and media power and its perception.