The ‘Wikipedia Dispute Index’: A Collaborative Seismometer?

Exploiting online networks in order to map and predict cross-linked social phenomena such as the commonality in consumption (Amazon), electoral behavior (Google), the spread of civil unrest (Twitter) or even the outbreak of diseases (Google and EMM) continues to be both an intricate and promising field of research.

Thus, it comes as little surprise that Gordana Apic, Matthew J. Betts and Robert B. Russell from the University of Heidelberg recently suggested an indicator for measuring geopolitical instability by applying the principle of association fallacy, primarily used in life sciences, to online encyclopedia Wikipedia’s entries whose impartiality is contested. In their brief research paper titled ‘Content Disputed in Wikipedia Reflect Geopolitical Instability‘, the authors accordingly argue that “quantifying the degree to which pages linked to a country are disputed by contributors” correlates with a country’s political stability. In other words: since instability is best represented by its underlying conflicts, the multitude and diversity of user-generated associations, namely: disputes, is believed to reveal some form of qualitative pattern that can be arranged as a stability ranking.

What may somewhat sound like a platitude and hasty induction at first glance, is elaborated more methodically by means of a so-called ‘Wikipedia Dispute Index’ (WDI) whose major benefit, as its inventors admit, is rather its simplicity than its objectivity. It is calculated as follows:

WDI = log (F_dispute/F_ave)

Where F_dispute is the number of disputed pages linked to a country (D) divided by the total number of pages linking to the country (N), and where F_ave is the average of F_dispute over all countries considered. Positive values thus denote countries with more disputes than average; negative values the opposite.

Applying this simple metric to a set of 138 countries/regions flagged as ‘neutral point of view’ (NPOV) dispute results in a quite foreseeable, but hardly impeccable projection of political instability with some of the ‘usual suspects’ (Middle East, Kosovo, BiH, North Korea etc.) coming to the fore. To better make sense of these findings, a weekly-updated and interactive visualization of the respective data sets is also offered, which, in my opinion, raises more questions than it answers. To substantiate this claim, a closer look at the model and the methodology is advisable:

• Only 138 of 497 countries/regions available at Wikipedia, according to the authors, provide “sufficient data to compute the indicator with confidence”, which means that – for the sake of the WDI – a country without significant squabbling is virtually non-existent. Obviously, the reverse argument (i.e. no dispute means perfect stability) cannot be valid though. Nonetheless, the statistical relevance of the findings must surely be questioned, all the more as the relevant values for Northern African and Middle Eastern countries remained stable over this year’s uprisings in the region.
  
• Another major, yet maybe necessary flaw of the research design is its exclusive reliance on the English-speaking version of Wikipedia. The problem here is twofold: 1. Is the sample used representative enough to draw broad conclusions (particularly on non-English speaking countries) and generalizations from relatively little evidence? And 2. Does this open the floodgates to a subtle selection bias which would distort the explanatory power of the indicator?
  
• “The evaluation of our ranking for the most cross-linked countries suggests that debates in Wikipedia correlate with regional instabilities all over the world,” Prof. Robert Russell of Heidelberg University’s Cluster of Excellence CellNetworks is quoted. Although it should go without saying, it cannot be stressed enough that correlation does not imply causation. The authors point this out when giving the example of  a Swiss regional football team whose Wikipedia entry is disputed because of a Moldovese player. Simply put: the fervor of Swiss football fans can hardly account for the political stability of Moldova.
• At last, guilt by association is a derogatory concept (as opposed to honor by association). Yet, many NPOV disputes may not imply negative reservations of a contributor towards a specific country, but rather towards another contributor’s preoccupations. Those disputes are mostly mediated and – ideally – resolved, at least for the short term. Only 7.8% of the entries disputed in 2007 were still disputed in 2010, raising the question if – qualitatively speaking – they really make for a solid indicator on the long run.
  

So finally, while the WDI is certainly worth being discussed and developed further, John Godfrey Saxe’s classic poem of ‘The Six Blind Men and the Elephant’ remains as meaningful as ever:

And so these men of Indostan
Disputed loud and long,
Each in his own opinion
Exceeding stiff and strong,
Though each was partly in the right,
And all were in the wrong!