What Percentage of Predictions Fail?

by Hubertus Hofkirchner -- Vienna, 11 May 2017

As Prediki's Chief Futurist, people ask me a certain question all the time: "What percentage of predictions of politicians, economists, or financial experts fail?" Even though there is a stunningly simple answer to this question, predictions are a highly complex subject, so the two digit answer needs some explaining. First, let’s forget crystal balls, Gandalf, and Harry Potter for a moment. Let’s use Prediction Science. There are three hidden levels to this question.

Level 1 - Simple Prediction

1. A forecast impacted by human action can never be 100% certain, because humans will react to forecasts with unforeseen actions which in turn can change the future dramatically, in accordance with chaos theory.

2. We can only measure the accuracy level and forecast bias of a specific method for multiple predictions - not a single one - with regards to specific forecast topics.

So: What level of unavoidable inaccuracy or bias do we chose to call a prediction failure? What is the commercial worth of predictions with more accuracy, less bias, or better reasoning compared to the monetary cost of producing it?

Level 2 - Decision Making

Human decisions combine forecasts with their (never comprehensively known) subjective value judgements. We enter the next level: Every human action has a purpose. The decision maker can act on an inaccurate or biased forecast but still achieve the actual purpose successfully.

So: Do we chose to call a success a prediction failure?

Level 3 - Deception Intent

Also, the question assumes a one-way causality, that politicians act on forecasts, implying that bad forecasts will cause bad decisions.

However, politicians (including top managers in corporate politics) often work the other way round. Let’s assume that a fictitious politician (or the lobbyist paying him) wants to trigger a war. He produces a forecast by which the citizens’ purpose will support the liberation (politically correct word for war) of some country, somewhere. For example, the politician might predict the existence of unspeakable weapons. The public diaspproves the weapons, thus approves the war. The politician's public approval rating soars in tandem.

Of course, when the fabricated prediction fails to materialise, the public purpose is frustrated.

So: Do we chose to call this a forecasting failure? After all, the politician’s real intent did work out perfectly fine.

The Riddle's Answer

The big answer is not 42, as claimed elsewhere.

It is 50%.

Why? There is no such thing as a forecast failure (see 1. above) but there certainly are right and wrong decisions (see 2. above), and “failure” is but the non-achievement of the decision maker’s true purpose (see 3. above).

Let's take a financial investment decision: Typically, an investor wants to make a return at least in line with benchmark, i.e. the average investor. On the stock exchange, it is obvious by tautology, that 50% of investors will perform above benchmark and 50% below. This very tautology is of course exactly valid for economic and political decisions. So the answer is: 50% of forecasts fail.

• Back to the Management by Predictions Blog
• Follow this blog for more stories, new case studies and how-to's.