Month: May 2009

In a time where numeric and statistical models of reality are in crisis, there are still people that think that re-packing expert judgment using a formula somehow makes the underlying assumptions more valuable. The Russian Center for Policy Studies, for example, offers what it calls “the Dow Jones of International Security”, an index that “is meant to demonstrate the extent to which the international security situation differs from the “ideal” (…) at each point in time.”

According to the Center, the index is based on a complex methodology that is characterized by “its comprehensiveness, robustness, and clarity.”

The following formula is used to calculate your security:

_{Screenshot: Center for Policy Studies site}

According to the Center’s methodology page, the factors above include “the threat of global nuclear war, the number and intensity of local conflicts, the type of political relations between various countries and international organizations, the intensity and scale of terrorist activity, the stability of the global economy, and the threat posed by man-made catastrophes and epidemics.”

Now the question is, how does the Center collect its data to calculate the security index?

“It is calculated on the basis of expert analyses of the probability of the occurrence of one or another global or regional event that would have a direct impact on international security. Each such event is given a certain score on the scale we have developed.”

So the index is basically based on expert judgment which is quite unreliable:

Philip Tetlock pointed out in his book “Expert Political Judgment: How Good Is It? How Can We Know?” that some 82’000 expert predictions that he tracked for two decades where only a tiny bit more reliable than random guesses or monkeys throwing darts at a board.

The problem with the index above also seems to be that perception of the current security situation is treacherous. For example on 10th September 2001 the West, especially the US seemed to be quite a safe place to most analysts. However, one day later the West was perceived as a side in a global war against a violent ideology. Nicholas Taleb’s “black swans” or random events with high impact, make an analysis of the current security situation much more difficult. It would be interesting to know how the Center’s experts had rated the probability of a global event one week before 9/11.

To be fair, the Center of Policy Studies is just doing what other political risk and even market analysts of big banks are doing: Selling their predictions by highlighting the value of expertise. As long as the marketing works and people really believe that accurate mathematical predictions of the future or even an accurate model of the present realities are possible, they will sell their products and the industry will continue to grow.