Hindsight is 2020, foresight is 2021. The COVID-19 pandemic abruptly changed the lives of almost everyone on the planet, causing more than four million recorded deaths, changing the way we travel, work, and socialize, as well as reducing the global economic output by trillions of dollars. As such, the pandemic has reinforced the willingness to engage in strategic foresight and to «think about the unthinkable».
However, what does this mean in practice? What foresight methods and tools are available, and which are suitable for what purpose? The interdisciplinary nature and breadth of strategic foresight makes it hard to provide comprehensive answers. Still, some methods, such as prediction challenges, deserve particular attention. Prediction challenges are a relatively new, Internet-enabled phenomenon. As such, they are not (yet) integrated into most governmental foresight processes. At the same time, the performance-based selection of online volunteers from prediction challenges has fared very well compared to other approaches in prize competitions organized by US Intelligence Advanced Research Projects Activity (IARPA).
How it works
A prediction challenge consists of a forecasting question, to which participants can submit their forecasts. Participation on prediction platforms is not restricted by demographics or credentials but is open to anyone with an interest in forecasting and/or the topic area. The primary incentive to participate is points or ranking systems that reflect the prediction record of users. Some platforms also offer awards to the best forecasters.
Answers to prediction questions can be binary (e.g., Will Xi Jinping be General Secretary of the Chinese Communist Party’s Central Committee on December 31, 2022?), multiclass (e.g., Which team will win the Major League Baseball World Series in 2021?), point estimates, or probability distributions (e.g., When will the first Artificial General Intelligence system be devised, tested, and publicly known of?). Participants can usually update their predictions before a challenge is resolved, and aggregate forecast timelines often show how the average answer on questions has evolved.
Notable examples
- The Good Judgment Project began in 2011 as part of a competition by IARPA. Good Judgement Inc. is the commercial spin-off from the project. It operates Good Judgement Open, a public forecasting tournament with questions ranging from geopolitics to finance, to sports. The company also provides access to its most successful forecasters as a service. To qualify as such a «superforecaster», an individual has to score in the top 2 per cent of the Good Judgement platform. Tetlock and Gardner have shown that the performance on Good Judgement contains clear elements of skill. The correlation of the performance between two years is circa 0.65 across all forecasters and about 70 per cent of the best performing forecasters in their Good Judgement Project were able to maintain their status year-to-year.
- The company Metaculus was founded in 2015 and offers a platform for probability predictions to binary questions, numerical-range predictions, and date-range predictions.
- The Forecast app lets people use virtual points to trade on future events and outcomes. In October 2020, Facebook’s New Product Experimentation team opened the app up to all users in the US and Canada.
- Foretell is a project launched by Georgetown’s Center for Security and Emerging Technology that focuses on questions relevant to technology-security policy. It has a particular focus on US-China politics, AI, and information technology.
- Cosmic Bazaar was launched in April 2020 by the UK government to supplement traditional forms of analysis and prediction. It includes 1300 forecasters from 41 different government departments and several allied countries.
Overall, prediction challenges are an interesting new tool that can identify and leverage individuals with high generic forecasting skills either within government administrations or more broadly within society. Persistently successful forecasters can be used as a slack cognitive resource to help answer questions in a crisis. Furthermore, prediction challenges may promote cognitive diversity, as pseudonymity allows a junior data scientist to contest the predictions and reasoning of a senior ambassador without regard to bureaucratic hierarchies. Prediction challenges appear particularly suited for forecasting technological and political events in the next five years. On longer time horizons forecasters do not fare better than chance on events, and it makes more sense to focus on trends or scenarios. Furthermore, there are some caveats concerning low probability, high impact risks, as well as regarding public relations risks.
For a general overview of strategic foresight with illustrative examples and links to documents with more in-depth discussions and instructions for specific methods, please read the CSS Risk and Resilience Report «Strategic Foresight: Knowledge, Tools, and Methods for the Future», commissioned by the Swiss Federal Commission for Nuclear, Biological, and Chemical Protection and the Swiss Federal Office for Civil Protection.