Risk Simulation Software

In the last two blogs I have challenged the idea that simulation results can be boiled down to a single statistic with any positive benefit. The context of a statistic is incredibly important, which is another reason why many statistics and charts/tables should be reported on, not simply one figure. And here’s a compelling reason why.

Consider two competing, similarly-sized projects, of which a company can only pursue one. Now let’s say this company would like to take on the project that has the “least risk”. If they are only familiar with generating the P90 for the total project cost they will be forced to select the project with the lowest P90. But what if the key drivers for exceeding the P90 are easier to mitigate in one project compared to the other? Perhaps the project with the lower P90 also has a higher P95 or P99 – this means the catastrophic failure is actually greater despite a lower P90 and is the mathematical equivalent of “when things go bad, they go really bad”. Not all P90s are created equally! Such an adverse outcome might sink a smaller company where a larger one could wear the loss. The context of the company running the analysis also impacts the context of the analysis itself.

So you can see not only do simulations generate results with which informed decisions can only be made if approached holistically, but if the language used is restrictive this outcome will never be achieved. Risk analyses are a necessary part of business because most of us wish to minimise the chance that something bad will happen, quite simply. Even if a manager tells you they “want the P90” what they are really asking is “tell me about the risk we’re facing”. The answer to this fundamental question is not found in a single figure taken from a simulation, but in a range of charts and tables which require correct interpretation.

More so, Monte Carlo simulation itself is only one piece of the risk and decision assessment pie. Decision modelling and optimisation, predictive modelling and statistical analyses should also form part of the quantitative approach to uncertainty. There is life beyond just risk simulation software, and I intend on exploring that in future blogs.

» Part 1
» Part 2

Rishi Prabhakar
Trainer/Consultant

Where I left off last time was lamenting the use of Monte Carlo simulation to create a single value (statistic etc.) from a model. It might still not be clear why this is anathema to me, so here goes.

A simulation is not a number. It’s not one possible (future) outcome – that’s a scenario. Monte Carlo simulation is a methodology for understanding one’s exposure to outcomes not situated close to the central tendency of the process/project in question. Note the plural “outcomes”. Risk analysis, when done properly, should let you know essentially all possible outcomes and how likely they are for your model. Output from a simulation can include a plot of means (over time), or P5s, or P95s, or the mean ± one standard deviation or any number of statistics. But that’s not plotting a simulation! Let’s not give a minimalist graph too much credit.

Such statements also perpetuate the idea that simulation is only used for creating means (or other centrally tending statistics) and ignores the wealth of information available. Risk simulation software exists to help you do risk analysis which must include not only several statistics but also sensitivity information. It is all too easy to turn a risk assessment into a hunt for a regularly asked for percentile (such as the P90) and there ends the task. I see this a lot, especially in project cost estimation where the pressure both from management and regulatory bodies is to accurately estimate some large percentile. Once found there is usually scant further risk analysis.

Nothing good ensues. When risk analyses are run “to get ‘the’ number” they become simply another box to tick in a process and ultimately any benefits (perceived or actual) will be forgotten and lost to the ages. The notion of context is also lost. No single number by itself really means anything, or at least shouldn’t mean anything to a decision maker. I have often heard phrases like “the model returned/gave $1.2m” followed by an audience nodding in agreement. Huh? Which statistic are you talking about there, and how about reporting a few other numbers around it to place that $1.2m somewhere meaningful?

In the next installment I will look further into this issue of context and hopefully prove the necessity of an holistic approach to understanding and reporting simulation results.

» Part 1

Rishi Prabhakar
Trainer/Consultant

New measures were published by the UK Government in November 2009 to protect communities at risk from ‘flash’ flooding when drains get overwhelmed by sudden downpours.  The Flood & Water Management Bill going before Parliament will give new powers to local authorities to manage surface water.  Environment Secretary Hilary Benn said, “We’ll see more severe weather as climate change takes hold, with heavier rainfall and potential flooding. The weather in the last couple of weeks has shown that this risk is very real.” 

Recent events in Cumbria in the UK suggest climate change really should be at the top of the agenda, and this new bill will certainly go some way to managing the risk associated with it.

Risk analysis has its place in determining the risk associated with climate change, and Palisade's risk modeling software has helped numerous organizations develop reports on likely outcomes. Cambridge University’s Judge Business School recently used @RISK to provide key input to the Stern Review on the Economics of Climate Change. Released in 2006, this report undertaken by the British government by Lord Stern discusses the effect of climate change and global warming on the world economy and is the largest and most widely referenced report of its kind.

They researched issues such as the impacts of the sea level rising and increases in temperature making land infertile or unfarmable, and balanced these against the costs of various options available to tackle global warming.  

At one end of the scale, doing nothing costs nothing, but the environmental consequences will be high. However, activity that reduces the severity of the impacts may itself be very expensive. The aim of the @RISK model was to enable people to make informed decisions on the optimum way to deal with climate change (ie how much to cut back on damaging activity and what methods to use).

@RISK risk simulation software also helped researchers tackle a key problem associated with investigating climate change, namely that the different effects of the various factors which influence it are themselves, undetermined. For example, the historical evidence does not pin down exactly how much global temperatures will increase if CO2 emissions double. @RISK enabled researchers to quantify this uncertainty in order that they have a measurement of the accuracy of their findings.

Risk analysis models should go some way to determining outcomes, and potentially help us prepare for the tragic events that have unfolded in recent days in the UK.

Craig Ferri
EMEA Managing Director of Risk & Decision Analysis