Reserve Estimation

Why is it that most of the high profile projects managed by the government in the UK all ultimately become beset by problems? A number of projects jump to mind – the Millennium Dome, Wembley Stadium and currently the NHS IT. All three have been plagued by developmental delays and financial mismanagement.

Recently, yet another worthy, but ambitious project has been announced – the North-South high speed rail line to connect London to Scotland. One wonders if the government undertakes detailed quantitative project risk analysis for its infrastructure initiatives?

A good example to highlight in this context is ENGCOMP, a Saskatchewan-based engineering consulting firm that has worked with the Canadian Department of National Defence (DND) to help define budgets for the fourth phase of construction of its Fleet Maintenance Facility at Canadian Forces Base Esquimalt in Victoria, British Columbia. Using @RISK, a Monte Carlo simulation tool, ENGCOMP helped the DND define and secure budget approval from the Federal Government’s Treasury Board. The consultancy firm was able to estimate the impact of the variability and uncertainties pertaining to risks, costs and scheduling. This assessment enabled it to estimate the project risk budget or the risk reserve and schedule contingency, which were both factored in when defining the total project cost of the infrastructure project.

The fact is, in the world of business, risk is inherent and unavoidable. Whilst one cannot completely control risk, one can certainly help reduce uncertainty, greatly increasing the chances of project success. For instance, a key finding of the project risk analysis conducted by ENGCOMP was that, taking into account all the risk and uncertainties on the project, there is an 85 per cent chance that the Fleet Maintenance Facility project will be completed in January 2014. A fairly positive result for the DND, given the scale and complexity of this project in question.

Craig Ferri
EMEA Managing Director of Risk & Decision Analysis

Where I left off last time was lamenting the use of Monte Carlo simulation to create a single value (statistic etc.) from a model. It might still not be clear why this is anathema to me, so here goes.

A simulation is not a number. It’s not one possible (future) outcome – that’s a scenario. Monte Carlo simulation is a methodology for understanding one’s exposure to outcomes not situated close to the central tendency of the process/project in question. Note the plural “outcomes”. Risk analysis, when done properly, should let you know essentially all possible outcomes and how likely they are for your model. Output from a simulation can include a plot of means (over time), or P5s, or P95s, or the mean ± one standard deviation or any number of statistics. But that’s not plotting a simulation! Let’s not give a minimalist graph too much credit.

Such statements also perpetuate the idea that simulation is only used for creating means (or other centrally tending statistics) and ignores the wealth of information available. Risk simulation software exists to help you do risk analysis which must include not only several statistics but also sensitivity information. It is all too easy to turn a risk assessment into a hunt for a regularly asked for percentile (such as the P90) and there ends the task. I see this a lot, especially in project cost estimation where the pressure both from management and regulatory bodies is to accurately estimate some large percentile. Once found there is usually scant further risk analysis.

Nothing good ensues. When risk analyses are run “to get ‘the’ number” they become simply another box to tick in a process and ultimately any benefits (perceived or actual) will be forgotten and lost to the ages. The notion of context is also lost. No single number by itself really means anything, or at least shouldn’t mean anything to a decision maker. I have often heard phrases like “the model returned/gave $1.2m” followed by an audience nodding in agreement. Huh? Which statistic are you talking about there, and how about reporting a few other numbers around it to place that $1.2m somewhere meaningful?

In the next installment I will look further into this issue of context and hopefully prove the necessity of an holistic approach to understanding and reporting simulation results.

» Part 1

Rishi Prabhakar
Trainer/Consultant

In Part 2 of this series I finished by asking what should be done with historical data, now that we have decided that storing it is probably a good idea. I won’t keep you waiting any longer.

Auditing and calibration of the model at both the micro and macro level. It’s as important as any other element of risk or statistical analysis, or indeed the model building itself. At the distribution level historical data helps to both parameterise the distributions and in fact select them in the first place. As a minimum a few data points will help you to understand possible central tendencies and variability for your risks, and also generate a list of feasible distributions to choose from. With a reasonable number of observations @RISK for Excel can be used to fit distributions to the data taking care of both distribution selection and parameterisation simultaneously. Only five data points are technically needed, but a reasonable fit will require either more than that or other holistic information to achieve validity.

At the macro level total project cost estimates are often ignored from the portfolio perspective. Commonly high percentiles are reported from such models to use in a ‘contingency’ calculation, such as the P90 or P95. Whilst a high percentile, the P90 (say) should still be exceeded 10% of the time! If your projects never go over this percentile then either there are some major mitigating factors not included in the model or the volatility is being consistently overstated. Likewise, the P10 for total cost (these ‘good’ percentiles are rarely if ever reported or considered in project cost estimation work) should be bettered in roughly 10% of projects. If this is not the case then the upside risk has been overstated. This may be due to misconceptions about the positive skewing present in most cost/delay risks or mistakes made in the parameterisation of the risks where the estimate (“most likely” etc.) is actually the “best case” or close to it, rather than a central tendency of the process over time. There could also be other possibilities.

No matter how you look at it, the collection and intelligent use of historical data is integral to effective and useful risk analysis and management, and critical to achieving valid Monte Carlo simulation results. If you aren’t currently recording everything you can get your hands on start right now!

 

» Part 1
» Part 2

Rishi Prabhakar
Trainer/Consultant

In my last blog I mentioned a ‘fact’ about data that came up during a recent public training course (Decision-Making and Quantitative Risk Analysis). This fact stuns me every time I think about it, and certainly floored me the first time I encountered it. So many companies just don’t have it.

Data, that is. Historical data from completed projects, sometimes billion-dollar projects, is simply not collected especially in resources and infrastructure cost estimation. Instead every risk is re-estimated from scratch in every new project based entirely upon an estimator’s recollections or guesses. This is not a suggestion that estimators don’t know what they’re talking about, rather that the benefits of adding historical data to the analysis far outweigh the cost of gathering the information in the first place.

I first worked in the banking sector, hence my surprise to learn of this lack of data storage in certain areas of risk analysis. Project cost estimation, especially in resources and infrastructure – I’m talking to you. In financial circles there are literally millions of data points collected daily across the entire organisation. Gathering data (and then analysing it for some benefit) is simply ‘what we do’, and this process isn’t challenged. Some of the data is quite ‘small’, such as the number of seconds a particular caller was kept on hold before being answered, and others are quite ‘big’, such as multi-million dollar losses due to fraudulent activities. Regardless, it’s all kept in the knowledge that information is power – in this case the power to make intelligent decisions in the future.

How can you judge the efficacy of an estimation process (workshops etc.) if you don’t track the final observed outcomes specifically to make such a judgment? Well, you can’t. And that leaves your company’s risk and decision assessment process in limbo. Without measurement there can be no process improvement or corporate learning. Are you ‘passing’ or ‘failing’ with your use of Monte Carlo simulation via risk analysis software?

Generally the observed outcomes for risks in models will be near the estimated value, and this is to be expected. However the main role of risk analysis is to adjudge exposure to the unexpected. Far too many cost estimation models have very little volatility in their line items. I am very curious to know just how often the realised value of a given line item is outside the range of “possible values” as defined in the model. And what about the total project costs overall? This hints at and leads to the big question which is what could/should be done with such data if it were to be recorded?

I shall address these questions in the next blog. I know you’re excited to find out!

Rishi Prabhakar
Trainer/Consultant

It’s clear that the financial crisis has exposed a number of failings in the practice of risk management. In my last post I talked about the relevance risk analysis and the disciplines of ‘quantitative risk management’ (QRM) and ‘decision making under uncertainty’ (DMU) are to all sizes of organisations, be it large or small. 

However, how accessible are these disciplines to the average size business across the globe today?

With the need to make more informed decisions more pressing by the day, thankfully QRM and DMU and now far more accessible than ever before.  Traditionally systems tended to be expensive, enterprise-based applications targeted at large companies who were prepared to spend considerable time, money and human resources.  The result was an all-singing, all-dancing product which often ended up underused due to confusion on the part of the very employees who were supposed to make it work.

Steady increases in computer processing have given the desktops of today as much power as the high-end servers of a few years ago, meaning that risk analysis and management is now an achievable goal for businesses of all sizes.  Palisades @RISK and Decision Tools Suite software are such desktop risk and decision analysis tools – working within Microsoft Excel and therefore being accessible to a large number of users.

‘Monte Carlo Simulation’, a technique originally conceived by scientists working to develop the atomic bomb as part of the Manhattan Project, is an inherent part of @RISK, a cornerstone of the Suite.  It enables users to introduce uncertainty into their previously static spreadsheets, which lets them look at things in a probabilistic, rather than a deterministic way.  In layman’s terms, this means that rather than companies and individuals making decisions based on estimates or best guesses, they can see all the potential outcomes to a venture – and how likely these scenarios are to occur.

For many companies this significantly improves the decision-making process.  Firstly it requires a change in the methodology of employees responsible for assessing risks and opportunities and secondly for the first time employees have a tool which allows them to communicate their recommendations to management or colleagues in a transparent and standardised way.  Equally, being able to look at scheduling risk in a probabilistic and quantitative sense allows for the allocation of labour and resources in a way which minimises slack and wastage whilst maximizing ROI.

So, it would seem that the new ‘risk management’ language that is starting to develop in the workplace and being taught to a new generation of managers on MBA courses should be welcomed.  With the accessibility of the technology available to assist them, we need to make sure that organisations do more than just pay lip service to QRM and DMU if they are to reap the rewards.

In my next blog I’ll be giving you the my top ten tips to adopting a health approach to risk, that will help businesses of all sizes maximize their risk management programmes.

Craig Ferri
EMEA Managing Director of Risk & Decision Analysis

Using this technique the researchers estimated that for every laboratory-confirmed case of H1N1, there were actually 79 cases.  In other words, predictions of a pandemic were not hysteria.  And while public health officials are unlikely to get hard data that will allow them to measure the actual extent and severity of H1N1 infections, no doubt without efforts to prepare for the virus, their rearview mirror methods would eventually tell a much grimmer story.    

The recent turbulence in the global economy has projected the word ‘risk’ into many everyday conversations, both commercial and personal: the unacceptable risks taken by fund managers which led to the collapse of major financial institutions; companies risking bankruptcy as a result of recession; the risk of people losing their jobs – and potentially their homes; and so on. 

As a result there is also increased talk of risk analysis, which in turn has brought disciplines known as ‘quantitative risk management’ (QRM) and ‘decision making under uncertainty’ (DMU) firmly into the business zeitgeist.  But for many small to mid-size companies, QRM and DMU are still regarded as something of a dark art and one that is not relevant to their day-to-day activities.

The truth is, that in boom or recession businesses make decisions every day – each with an associated level of risk.  Much of this decision making is undertaken by looking into the issues facing a business, putting some numbers on them to calculate their impact, and then mitigating or allowing sufficiency contingency in the event that things go wrong.

Examining business decision-making in detail shows us that most businesses could benefit from making the link to risk analysis, and from there taking a more strategic approach to the discipline. Cost estimation, budgeting, cash flow forecasting, operational risk assessments, sales forecasting – in fact any part of a business where there is uncertainty can all be made more robust and meaningful.

Recession has brought the idea of QRM to the forefront of business owners’ minds.  Essentially it is a valuable aide to making better, more informed decisions where the amount of uncertainty on which they are based is known. 

Risk analysis is no longer a dark art, but in today’s economic climate, is an essential part of the business decision-making process, no matter what size the organisation.

In my next blog we’ll look what technology is available today that will help businesses making better decisions now and in the future.

Craig Ferri
EMEA Managing Director of Risk & Decision Analysis

I'll defer to Mr. Miller, whose blog is loaded with interesting details.   

At November's supercomputing conference in Portland, Oregon, IBM announced that its researchers working with a team from Stanford University had succeeded in developing an accurate simulation of human brain function. The simulation will be capable of emulating sensation, perception, action, interaction and cognition.

On Thursday, December 3, 11am-Noon ET, Jay O’Connor will present a free Live Webcast about project risk management.

A project risk analysis is only as good as the model that was used to prepare it. It is critical that the model be constructed to reflect the risks specifically associated with the project. The model must be able to accurately reflect the risks associated with schedule, quantities, cost and the residual unmitigated risk items from the qualitative risk analysis. The model should also take into account the interrelationships and dependencies of these items.
This webcast will address these issues and present examples of how results can vary based on the level of detail used in preparing the risk analysis, and will include the use of @RISK, and @RISK for Project.

Palisade is pleased to host Jay O’Connor’s presentation. With over 25 years of experience in the areas of estimating, planning and quantitative risk analysis for international projects, Jay understands the complexities that are associated with identifying and assessing project risks. His experience includes both the owner’s and contractor’s side of engineering and construction projects. He has worked in the upstream and downstream oil and gas industry sectors and the pulp and paper sector. His career has taken him to the United Kingdom, Japan, Indonesia, Malaysia, Singapore and Australia.

» Register now (FREE)  
» View archived webcasts

In a previous blog, I presented a very simple way to allocate contingencies to uncertain cost elements in the project risk management process. However, that methodology works well when there are not risk events that affect a cost element or a group of cost elements.
A risk event is described by two elements: the probability of occurrence and the conditional impact to the project given its occurrence. For example, we have a risk that describes the possibility of a new regulation. If it occurs, it will increment the cost of group of cost elements by a minimum of 10%, most likely 15%, and a maximum 20%. If the risk does not occur, no impact will be observed. Using a Discrete and a PERT distribution, we can model such risk such as:



When sampling from this distribution approximately only 20% of the time will generate a multiplier with a minimum of 1.1, most likely 1.5 and a maximum of 1.2; in 80% of instances the multiplier will be 1. That means that only 20% of the time the risk will increment the cost of selected cost elements by the multiplier previously described as show in the figure below:



In addition to risk events in our cost risk analysis models, we often use distributions that describe cost uncertainties. These distributions model ranges are mostly in a different order of magnitude. Therefore, the variance will also be in a even greater order of magnitude. For example, the cost of Item 3 modeled using a 3-point estimate (i.e., min 100,000, ML 120,000, and max 150,000) has a variance of   87,698,412.70), while the variance of the risk event is 0.0036. 

If we are to distribute the contingency using the % of contribution of the variance method, the risk event that we just modeled will be ignored even though we know that such risk event has an impact that we cannot dismiss. Given this practical scenario, the method of variance contribution will not work appropriately.

As an alternative, we can use a tornado diagram that results from @RISK’s sensitivity analysis. Here we can use the regression coefficients to understand what risk events or uncertainties are affecting the total cost in a more drastic way. In the case that you also incorporated events that represent an opportunity to reduce cost, you will observe that the coefficient is negative; in your allocation calculations you should not consider negative coefficients.

In the figure below you can observe the Regression Tornado. Here risk events and uncertainties are represented in a scale that goes from 0 to +/-1:



Knowing the regression coefficient of each input that affects the total cost in a negative way, we can construct a table and obtain a normalized percent that can be used to distribute contingency. If for example, we have a contingency of $100,000, it can be distributed to each input proportionally to the regression coefficient as shown below.



Some risk management experts do not distribute the entire amount of the calculated contingency. It is common practice to distribute only a percentage of it (i.e., 70%). The remaining amount will be used as a reserve that will handle unidentified risks.

Javier Ordóñez, Ph.D
Director of Custom Solutions

In a previous entry to this blog I discussed how to assess the contingency required in a cost risk analysis study. The next step is to allocate the calculated contingency to uncertain cost elements that drive the variation in the total cost of the project. In this way, the contingency can be better managed and controlled throughout the life of a project.

While reviewing literature on this topic, I found a practical way to do this. This methodology uses the percentage contribution of each uncertain variable (usually 3 point estimate distributions) to the variance of the resulting distribution of the total cost.

To apply this method, we need to report the variance of each input distribution and the variance of the end result. In case that input distributions are independent from each other, we can just add up individual variances to estimate the variance of the total. However, this is hardly the case since correlation between input variables is expected in cost models.

@RISK allows reporting statistics from an input distribution without running a simulation as well as statistics that describe an output. These functions are from part of the @RISK functions library: Statistic Functions> Theoretical and Statistic Functions>Simulation Results, respectively. These functions can be accessed using the fx icon from the @RISK toolbar. 

To report the variance of input distributions we can use the RiskTheoVariance and for the output RiskVariance. The construction of the allocation model is shown below.



In the project risk management model above, it can be observed that the % Contribution to the Variance of the Total Cost is calculated as a proportion of the input variance to the total variance. Once these percentages are determined we can use them to allocate the management contingency to each cost element. It can be also observed that the engineering allowance is also calculated, and the decision maker now has criteria to manage and control contingencies.

Javier Ordóñez, Ph.D
Director of Custom Solutions

Having successfully dodged high school physics I would not normally be sucked in by an article on quarks, but this one involved neural network computing, which I do understand pretty well.

When performing a cost risk analysis study, one of the key results is the amount of extra monetary resources that is to be added to the project cost baseline to guarantee that the budget is not exceeded at a certain confidence level. Good project risk management strategies must take this into account.

After defining the uncertain variables and risk events that affect the cost performance of the project, we can run a Monte Carlo simulation with @RISK to find out what the range of the total project cost is.  Simulation results can help us to explain the risk exposure that we have in the total cost of the project. The most popular statistics are the mean (average cost), the most likely cost, and the 10th and 90th percentiles.




To determine the contingency to be allocated to the project, we need to define what confidence level we would like to achieve: The higher the contingency level, the larger amount of contingency needed. For example, in the figure above, we are reporting the total cost of the project. Here we can observe that we are showing the 85th percentile that corresponds to a total cost of $7.8M (right delimiter).  We can say that there is only a 15% chance that we will exceed $7.8M, or alternatively, we have an 85% chance that the total cost will be less than or equal to $7.8M.  In the same figure we can also see that the 90th percentile of the total project cost is $8.02M.  We can say then that in order to increase our confidence level from 85% to 90%, we will need to add $220,000 to the total cost.

The calculation of the contingency is then accomplished by using the base cost estimate (BE) before the risk analysis was implemented, and the expected cost (EC) of the simulated results.

Some practitioners separate the contingency into two components: engineering allowance, and management contingency.

Engineering allowance (EA) is the difference between the expected cost and the base estimate:

EA = EC – BE

Management contingency (MC) is calculated using the difference between the cost at certain confidence lever (Cp) and the base estimate:

MC = Cp – EC

In our example, our BE = $6.5M; therefore, engineering allowance EA = EC – BE = 6.86M – 6.5M = $0.36M. 

For the calculation of management contingency, we use a confidence level of 85% so Cp(85%) = $7.8M; therefore, MC = Cp – EC = 7.80M – 6.86M = $0.94M.

In many situations, the suggested contingency might be excessive, so the need for a mitigation study is necessary. We can use the sensitivity analysis tool in @RISK to detect the key drivers affecting our total cost. This is valuable information so that we can concentrate our efforts in reducing the impact of risk events and uncertainties to the total cost. Below, we see a tornado graph with the most important drivers. The analyst will then explore the appropriate mitigation strategies and assess their implementation cost. A second simulation can be run to assess the effectiveness of the proposed mitigation plan, and compare the pre-mitigated and post-mitigated cost distributions.




In following blog posts, I will explain how to distribute the assessed contingency to cost elements and identified risk events in project risk management models.

Javier Ordóñez, Ph.D
Director of Custom Solutions

Confidence intervals are the most valuable statistical tools available to decision makers, and according a recent Six Sigma IQ article written by Dr. Andrew Sleeper of Successful Statistics, they are not being used as frequently as they should. Sleeper’s article  Have Confidence in Your Statistical Analysis!: Learning How to Use Confidence Intervals does an excellent job illustrating why point estimates are useless for making decisions, and how to determine what is the best confidence interval to use. Is it 90%, 95%, or some other value?

The article does not discuss how to calculate confidence intervals, since widely available software (for example, Palisade’s @RISK and StatTools) automates this task. Formulas and calculation methods are well documented in many books.

One example that Dr. Sleeper uses to illustrate his point: Suppose the CEO has decreed that we need CPK to exceed 1.50 for all critical characteristics. If I measure a sample of parts and announce “CPK is 1.63,” this sounds like good news. But then you ask a really good question: “How large is the sample size?” If you discover the sample size was only three, should you be worried? What if you discover the sample size was 300?
We have to make a decision about the capability of the population, but once again, the point estimate is not enough information by itself to make this decision. It is another useless number.

Instead, suppose I said “I am 95 percent confident that CPK is at least 1.52.” Or I could say “I am 97 percent confident that CPK is at least 1.50.” Either of these would be a true statement. And since sample size is used to make these calculations, they provide all information necessary to make the business decision.
These one-sided confidence intervals are often called lower confidence bounds, because the upper limit of each confidence interval is infinity. In the case of CPK, we usually don’t care how large it is, so a lower confidence bound is more appropriate than a two-sided confidence interval.

Because they are single numbers, point estimates are almost always above or below the parameters they are supposed to estimate. Without additional information, point estimates are useless for making decisions. But confidence interval estimates are very likely to be true, and the confidence level specifies and controls the probability that the interval estimates are true. Since properly applied confidence intervals incorporate sample size and other tested assumptions, these are reliable tools to make business decisions.

In addition to this article you can find alot of great information at Six Sigma IQ 

When modeling risk events, it is common that several events could affect the same cost element of a project. During the simulation, two or more risk events can occur at the same time. The question becomes how to calculate the total impact. This type of modeling technique is very common and often needed in project risk analysis, contingency and mitigation studies, reserve estimation and production forecasting.

A common practice is to aggregate the total impact. However, this simplistic approach might not be correct since it ignores the compounding effect of multiple occurrences. For example, if we have two risk events occurring and their respective impacts are a 20% and 40% cost increase, an additive model will calculate the total impact as 20%+40%=60%. On the other hand, a multiplicative (compound) model will calculate the total impact as 1.2 x 1.4 = 1.68; here the total impact is 68%. In other words, the impact of these risk occurrences will be greater than the summation of the individual impacts, which in many cases makes sense.

Using the table below we will show how you can use @RISK (risk analysis software using Monte Carlo simulation in Excel spreadsheets) to can calculate the distribution of the total impact that can affect the cost of a project activity. You can see that in this example we also model the opportunity of a cost reduction.





To model the % cost increase and reduction we use a Triangular distribution. Here we say that the maximum impact is the value in column F (the user can use other distributions or parameters as needed). With this logic the distribution of the impact of Risk 1 looks like:



and the total impact distribution results in:



The following table contrasts the results obtained using a compound and an additive model:



Something to note in the table above is that in the opportunity side, the maximum cost reduction using the compound model is less conservative than the additive model.

As seen here, coding this type of model is not difficult. My suggestion is that when you have multiple occurrences of risk events you explore better alternatives than the additive model. As always, your comments are more than welcome.

Dr. Javier Ordóñez
Director of Custom Development

Jay O’Connor is a Director at Turner & Townsend Inc. With over 25 years of experience in the areas of estimating, planning and quantitative risk analysis for international projects, Jay understands the complexities that are associated with identifying and assessing project risks. His experience includes both the owner’s and contractor’s side of engineering and construction projects. He has worked in the upstream and downstream oil and gas industry sectors and the pulp and paper sector. His career has taken him to the United Kingdom, Japan, Indonesia, Malaysia, Singapore and Australia.

Jay will present a case study at the 2009 the 2009 Palisade Conference: Risk Analysis, Applications, & Training. The conference is set to take place on 21 - 22 October at the Hyatt Regency in Jersey City, 10 minutes by PATH from Manhattan's Financial District.

See the abstract for his case study below, and see the full schedule for the Conference here.

Integrated Project Risk Analysis

When conducting project risk analysis, it is not uncommon for the qualitative risk, quantitative schedule and quantitative cost risk analysis to be conducted separately and kept independent of each other. While some software packages attempt to integrate all three into one analysis, these efforts tend to fall short in one area or another. Turner & Townsend’s approach is to integrate the residual risks and opportunities along with the results from the schedule risk analysis into the cost risk analysis to develop a more fully integrated project risk analysis. The presentation will discuss our approach to risk analysis.

October 21-22 in NYC

Building on the success of last year’s record-breaking event, the conference will offer a wide range of software training, model building, and real-world case study sessions. Last year, the event drew over 150 practitioners and decision-makers from a broad spectrum of industries. The @RISK and DecisionTools software tracks were more popular than ever. This year, we’re expanding software training with sessions that let you walk through examples and try the tools directly. This will enable you to take some new tips back to the office. Please join us in October for a great opportunity to learn and connect with colleagues.