IT projects are notorious for going off track and costing much more than estimated. But how bad can it get?

Following is a sobering quote from the book “How Big Things Get Done” by Prof. Bent Flyvbjerg and Dan Gardner

“My database revealed that information technology projects have fat tails. To illustrate 18 percent of IT Projects have cost overruns above 50 percent in real terms. And for those projects the average overrun is 447 percent! That’s the average in the tail, meaning that many IT projects in the tail, have even higher overruns than this. Information Technology is truly fat-tailed!”

Prof. Bent Flyvbjerg and colleagues have curated the largest database of project cost overruns across industries. They have collected and analyzed a dataset of 5392 IT Projects to examine the probability distribution of IT project cost overruns. Cost overrun is defined as a ratio.¹

Cost overrun = (actual cost)/(estimated cost)

The distribution of projects studied are as follows- 2739 IT projects across 872 firms and 2653 IT projects undertaken by 104 US government agencies. The projects in the dataset varied by type, with the most frequent being Supply Chain Management (25%), ERP (19%), and HR management (9%). The researchers also split the dataset based on project size and empirically discovered that the cost overruns exhibit the power-law tails irrespective of the project size.

The researchers did not distinguish between agile IT projects vs. non-agile IT projects and think that further empirical research is warranted to compare performance and respective probability distributions.

What are fat-tailed distributions?

The Probability Moocs mini-lectures by Nassim Nicholas Taleb on YouTube is a highly recommended series of mini-lessons in fat-tails and statistics. Bookmark it!

These are some fundamental statistical properties of a dataset:

• N = number of observations
• Xi = a Observation
• X̅ (mu) = mean
• X̃ (M) = Median
• σ (sigma) = the Standard Deviation

First, consider the Normal Distribution or the Gaussian distribution. It is also called thin-tailed distribution or the bell curve.

This means that when the data set is thin-tailed, the chances that you will observe a value ( X_i ) that is more than one standard deviation from the mean is only 15.8%

Imagine putting your hand in an opaque bag and then drawing a ball at random with a number on it. If the number on the ball corresponds to the height of a 20+-year-old Male in the United States, then you can expect the value to be closer to ~ 175 inches. And the chances that you will draw a ball with 10 inches or 1000 inches will be non-existent.²

The Height ( Xi ) dataset of Males in the US is a thin-tailed or Gaussian distribution.

Now, consider a dataset that is fat-tailed. In the fat-tailed distribution, a small number of outlier observations represent the bulk of the dataset’s statistical properties.

In a Fat-tailed dataset, “the body” is rising. 

The number of observations within one sigma of the mean will be much greater than in a Gaussian (> 68.2%), and the tail will be longer. The more observations you make, the more likely you will find an observation six sigma away from the mean. These observations far away from the mean will be infrequent but will be significant in determining the statistical property of the distribution.


Again, imagine putting your hand in an opaque bag and drawing a ball at random with a number on it.

This time, the number on the ball corresponds to the net worth of individuals aged 35-44 in the United States. You could be drawing a lot of balls, most of which are six figures or less, until you pick Mark Zuckerberg, whose net worth presently is over twelve figures—a 10^6 multiple of the median. The mean (average) of the net worth distribution is around ~440K³, which is 4x the median, and the skew in the dataset is caused by fewer huge values. The mean and median are not the same in a normal distribution but closer together.

A fat-tailed distribution has a long tail, and when the tail has a power-law relationship, you do not know how long the tail is. In other words, no number of sampling observations will give you a stable population mean. Depending on the sample you draw from the bag, you will get a different sample mean until you get the one significant outlier that destroys your assumptions about the values you should expect to draw.

Compare histograms of a randomly generated Normal Distribution {Blue} and a randomly generated Cauchy Distribution (fat-tailed) – {Red} with the same Mean (mu) = 0, σ (sigma) = 1, and N = 1000.

These statistical properties of both distributions are the same, but the observation values Xi behave differently.

F(x): Why does this matter?

“Don’t cross a river if it is four feet deep on average.” – Nassim Nicholas Taleb.

In the Gaussian, there is regression to the mean, whereas in the fat-tailed, there is regression towards the tail. In the fat-tailed, the average is misleading. Instead, you must watch for extremes and protect yourself from their impact to survive.

Now let’s re-examine the statement again –

“My database revealed that information technology projects have fat tails. To illustrate 18 percent of IT Projects have cost overruns above 50 percent in real terms. And for those projects the average overrun is 447 percent! That’s the average in the tail, meaning that many IT projects in the tail, have even higher overruns than this. Information Technology is truly fat-tailed!”

Here is a way to visualize the results of the research.

Imagine a fair six-sided dice—the board game variety.

When you roll the dice and get [2,3,4,5,6] – 83.33% of throws (for large N), your project’s cost overrun is less than 1.5 times the estimated cost. In some cases, there is no cost overrun or, in some instances, under-run.

But when you roll a one, which is likely for 16.66% of throws (for large N), your project costs at least 1.5x the estimated cost, and the average cost overrun is 4.47x the estimated cost, sometimes even 20x or far more. The maximum cost overrun in the research data is 280.4x  the estimated cost!

Let’s play a much-simplified game.

Say there are no cost overruns (actual/estimate = 1) for six-sided dice rolls that result in a [2, 3, 4, 5, 6] – (83.33% of rolls) and a 5x cost overrun when you roll one on the dice (16.66% of rolls).

The game’s player is an IT Executive with a budget of $100 in their portfolio and bets $10 for each roll of the dice. 

Here is an example roll. Say the player throws the dice and rolls a three. Since it is one of [2, 3, 4, 5, 6], the cost overrun is 1, and therefore has $90 left. 

In the next round, if the player rolls a one, the cost overrun is 5x, and they are left with $90 – $50 = $40. 

Each roll in this game represents a project with a cost estimate of $10. Then, depending on the outcome of the dice roll, a cost overrun factor is applied to calculate the remaining budget in the portfolio.

To keep things simple 

(A) We are not modeling the benefit gained from project completion. Typically, in IT departments, the Executive is judged on their budget performance and not on the business benefit performance. 

(B) Also, we keep only one project active at a time. This means we know the actual costs to calculate the remaining budget and proceed to the next round.

Q: How many rounds can a player be expected to last if each bet is $10?

A: In the best case scenario, ten rounds, when they never roll a one, and in the worst case, the player exits the game in two rounds because they rolled a 1 in each of two throws.

Q: Can a player end up with a budget deficit?

A: Yes, the maximum deficit can be -$40; this can happen when the remaining budget is $10 and the player rolls a 1.

Consider that 10 IT Executives are playing this game. Their objective is to have a career and to last through the year without blowing up their budget of $100. Each dice roll costs 10% of the budget, and the cost-overrun factor is either 1x (No overrun) or 5x (average of the tail from research), depending on the random roll of a six-sided dice. 

Following is a trial run 3D Histogram for ten players, their Remaining Budget, and the number of rounds each player remained in the game. 

In this example run, there are two winners. They last ten rounds and have a $0 deficit. And the biggest loser is a player that lasts ten rounds but ends up with a deficit of -$40.

Now let’s dial up the number of players and see the results when 1000 players (IT Executives) are playing this game.

Because we now have 1,000 players, we see more areas where a player can land. We also see “steps” in probability space because of the constraint parameters set in the rules for the game:

• Each project is estimated to be 10% of the overall budget of 100.
• An IT Executive exits the game when the remaining budget goes below zero, which also constrains the maximum downside to 1.4x of the IT budget.

In the real world, IT executives have more than one Project in-progress in their IT Portfolio. Yearly, they approve many projects to start/continue and will not know until later about the actual cost overruns. This means that with parallel Projects, the maximum downside can be significantly higher than 1.4x the budget. CEOs or the Board of Directors often terminate Executive careers before it worsens significantly.

Simulate variable cost estimate per Project.

Now that we understand the simplified game and know what to look for in the 3D Histogram, we will simulate variable costs for each roll.

Almost all IT departments have extensive approval and sign-off processes when the estimated cost of a project is over a threshold. Big projects attract Executive attention and governance oversight when projects are evaluated to cost beyond a threshold.

To simulate this policy, we will relax the constraint that every project must be estimated at 10% of the IT budget. Instead, for every roll for every player, the cost per roll is randomly selected to be any of {1, 2, 3, …,10}.

This means the company has a policy of never approving a project representing over 10% of the IT budget. But there can still be some projects that are at the 10% of budget threshold.

The cost-overrun is still a function of the estimated cost, and for 5/6 rolls of dice, there is no cost overrun, and has a 1/6 chance of 5x actual costs.

Following is a 3D Histogram for a simulation run with ten players.

You will notice that the number of rounds the winner plays exceeds 10. This is because some rolls are cheaper, and the corresponding cost overrun penalty removes less from the remaining budget. 

However, ten players are not sufficient to visualize all the spaces where outcomes could unfold. So we simulate for 1000 players.

Some observations: 

• The highest density of the player population plays more than ten rounds, with losses under -20.

In this simulation, allowing for smaller projects (estimated cost) significantly increases the number of rolls and reduces the likelihood of maximum losses.

• Some players experience a loss greater than $40.

The unluckiest player (not represented in this simulation run) could never roll a one and always bet $1 to last for 99 rounds, and then on the last round, rolls a one and bets $10 ($9 on borrowed money) to end up with -$50 in deficit.


This is the unfortunate reality of fat-tailed distributions. A single outlier event can wipe out your success in all previous rounds.

Reviewing the density of player outcomes, we can conclude that a portfolio of small bets outsurvives big bets.

The HBR Article, Why your Project May be Riskier Than You Think, 2011 provides some high-profile examples and makes a compelling case for Executives to prepare and stress test for significant cost overruns and benefit shortfalls.

Player Skill and Silent Graveyards

In the simulations above, players do not have the agency to apply their skills. This is because the roll of the dice is randomly generated, and the project cost is randomly selected to be under the max threshold. Therefore, we simulate 1000 players to implicitly simulate multiple “sub-strategies” that different player paths take.

We find that a portfolio of smaller-sized projects outlasts a portfolio of large projects by a significant margin. For another comparison, again for 1000 players, review the cases below.

Case (a) Min bet = 1, Max bet = 5
Case (b) Min bet = 5, Max bet = 10

Case (a) significantly outperforms Case (b). 

A good overall strategy is to get some returns for the smallest possible bet. All other gameplay strategies are subordinate to it. The simplest thing an IT department can do to improve the chances of portfolio survival is to reduce the project size.

An IT Executive or Portfolio Manager who assumes that project cost overruns follow the Normal Distribution will grossly underestimate the risk they are taking. This belief is expressed in everyday language as claims that cost overruns will all “wash out in the end.”  If you were to ask a gambler who frequents casinos, they would also say something similar. “You win some, you lose some.” 

This is because everyone who went bust can no longer afford to go to the casino or is not in the mood to answer your survey. Survey-based research papers on cost overruns run into the same fundamental problem. The graveyard of silent evidence is full of IT Executives who thought they were doing reasonably well until that big bad project blew up in their face. This is why excluding outliers in research papers leads to the mischaracterization of the risk in the domain. 

By analyzing actual vs. estimated costs across a large data set of private and public IT projects, the conclusion is that the domain of IT projects is fat-tailed and not thin-tailed. This means that you will have many observations closer to the mean. Still, the few outlying ones will be so extreme that they can wipe out all possible benefits your company has accumulated over its lifetime.

“What the power law of IT project cost overruns tells us is that extreme IT project cost overruns will occur. We cannot predict when the next one will happen or how big it will be, but we can predict that (1) more will happen and (2) sooner or later there will be one that is larger than the largest we have seen so far. Therefore, following the power-law logic, it will be no surprise if a large, established company fails in the coming years because of an out-of-control IT project.”¹

I recommend you review this paper and pick up a copy of the book to dig deeper.

Smaller is better

In non-agile IT departments, there is a common concern that we cannot get something meaningful for businesses with small project budgets. As a result, they take on bigger projects because they know their bureaucratic processes and need the project to be huge to allow for built-in inefficiencies of siloed work efforts.

Agile practitioners know that big projects fail exponentially bigly. We emphasize incremental development, testing, and integration of small batch requirements (stories) for the project in short cycles. Instead of inspecting for quality at the end, the quality of the project deliverables is maintained throughout the project lifecycle. Validating that working software delivers on expected business benefits and inspecting and adapting changes to requirements throughout the project life-cycle.

Digital Transformations are one of the riskiest efforts a company can embark on. And in many cases, competitors are closing the gaps on the incumbents at a frightening pace. This leaves little choice but to aggressively pursue digitization efforts. But the way forward is not to attempt big transformation projects. Instead, it is to break down large initiatives into smaller integrated projects focused on enabling business outcomes and corresponding changes to digital assets.

IT needs the capability to integrate and test software solutions in small increments, and the Business needs to break down complex initiatives into smaller projects so the value can be realized incrementally. The agile movement has paved this trail with frameworks, techniques, and practices. However, it requires frequent collaboration between the business and IT. The necessary skill needed for success is to incrementally realize business benefits so the business and IT can iterate towards integrated solutions that can evolve with changes in the business landscape.

Relabeling Projects to Epics and keeping the waterfall of large requirements scope flowing down to silo-ed development and testing will not work.

The change is to embrace the Business and IT relationship as a partnership truly. Your success with Digital transformations will depend on the Executive’s ability to craft small-size projects that deliver tangible business results.

Ps: Simulations were generated on Mathematica. It is addictive!

pps: Thanks to Michael de la Maza for reviewing the article and for feedback.

Footnotes

1. The Empirical Reality of IT Project Cost Overruns: Discovering a Power Law Distribution is a research paper by Bent Flyvbjerg, Alexander Budzier, Jong Seok Lee, Mark Keil, Daniel Lunn & Dirk W. Bester. [ Link to research paper ]

2. CDC NCHS Vital and Health Statistics, Series 3, Number 46 – January 2021. Table 11, page 15. [ Link here ]

3. Federal Reserve, Survey of Consumer Finances (1989-2019) [ Link here ]