Exploration and Production

In my last blog entry I introduced the notion that optimal decision making wasn’t ‘on the radar’ for many clients in Australasia, and laid out a couple of ideas why. I too once focussed on Monte Carlo simulation rather than decision evaluation, but last year the most obscure event changed that.

Call me a nerd of you will, but I like modelling problems in Excel. There is skill involved in setting up a problem such that the model assumptions aren’t too gross, and an art to making the model elegant. This elegance can be very important to optimisation problems, but more on that later. My first homemade optimisation problem was generated by motorcycle racing! MotoGP, to be precise. A friendly tipping competition with friends was formed at the start of the 2009 season with the following structure:

• Entrants played the role of Team Manager.
• Team Managers had a fixed budget to spend on riders.
• Either a few good riders could be purchased, or many lesser riders, or something in between.
• The team that had accumulated the most points at the end of the season was the winner and received kudos!

Although the future results could not be known of course so I set up and ran the optimisation with Evolver after the event to see what the optimal team selection would have been. Historical data could have been used to discover the type of rider mix that tended to be optimal and thus make an informed decision for this competition. The risk in having only a few riders was that any misfortune would have a big negative impact on the points won, whereas a team consisting of many (cheaper) riders was less likely to suffer such a fate. This downside scenario will be modelled into the 2010 MotoGP Team Manager predictive, optimised model (currently in production)!

What has this to do with the corporate world? Replace “team” with portfolio and “riders” with “assets”, “shares” or “projects” and you have a classic portfolio optimisation model. I hadn’t created this model with business applications in mind but I realised that was precisely what I was doing. An instant later I realised just how useful Evolver would be in many decision scenarios even though it doesn’t incorporate uncertainty (RISKOptimizer does).

In the next instalment I will further explore some practical applications for Evolver and you’ll see just how universally appropriate it can be.

» Making Optimal Choices, Part 1

Rishi Prabhakar
Trainer/Consultant

Something has troubled me for some time regarding the choices being made in risk land. I train and work with many clients whom have adopted Monte Carlo simulation techniques (via @RISK for Excel) into the day-to-day running of their businesses. By doing so they (hopefully) now have a good understanding of the exposure they are facing be it in project cost estimation, discounted cash flow analysis or, well, anything really. But this is only one facet of risk and decision assessment, specifically dealing with the descriptive statistical output from a simulation. What of the decision evaluation component? Why aren’t more of my customers analysing the decisions they make, or better yet actually optimising them? I have a few ideas why.

If you’re in business you have to make decisions. Big ones, little ones, yes/no, multiple state and continuous value decisions. Decisions that impact other decisions in simple or complex dependency structures. But are you making the best decisions possible? I’m sure important decisions aren’t being made completely randomly (I hope!) but I see many companies who rely completely upon qualitative techniques for their decision making (experience, gut feel, etc.) which of course means optimality is no more than a hoped for outcome rather than something that is actually being worked towards.
Firstly the decision model must be identified and then quantified, and this can be a difficult task. There is a level of modelling aptitude necessary for effective modelling that goes beyond merely knowing Excel and its functions, and into the construction of logical mathematical descriptions of possibly complicated processes. Relevant decisions need to be identified and the impact of those decisions combined into a formula that can be mathematically optimised. A critical component to all this is the knowledge that spreadsheet models can actually be optimised, and that in cases where Excel’s Solver fails there are Palisade products (Evolver and RISKOptimzer) that can perform optimisations under virtually any circumstance.

I too used to focus on Monte Carlo simulation rather than decision evaluation, and this was mainly a product of the clients I was dealing with almost exclusively when I first worked for Palisade. In my next blog I’ll tell you why that changed and also get a little more into the nuts and bolts of optimisation.

Rishi Prabhakar
Trainer/Consultant

Rumors of this death are greatly exaggerated.  

In the last two blogs I have challenged the idea that simulation results can be boiled down to a single statistic with any positive benefit. The context of a statistic is incredibly important, which is another reason why many statistics and charts/tables should be reported on, not simply one figure. And here’s a compelling reason why.

Consider two competing, similarly-sized projects, of which a company can only pursue one. Now let’s say this company would like to take on the project that has the “least risk”. If they are only familiar with generating the P90 for the total project cost they will be forced to select the project with the lowest P90. But what if the key drivers for exceeding the P90 are easier to mitigate in one project compared to the other? Perhaps the project with the lower P90 also has a higher P95 or P99 – this means the catastrophic failure is actually greater despite a lower P90 and is the mathematical equivalent of “when things go bad, they go really bad”. Not all P90s are created equally! Such an adverse outcome might sink a smaller company where a larger one could wear the loss. The context of the company running the analysis also impacts the context of the analysis itself.

So you can see not only do simulations generate results with which informed decisions can only be made if approached holistically, but if the language used is restrictive this outcome will never be achieved. Risk analyses are a necessary part of business because most of us wish to minimise the chance that something bad will happen, quite simply. Even if a manager tells you they “want the P90” what they are really asking is “tell me about the risk we’re facing”. The answer to this fundamental question is not found in a single figure taken from a simulation, but in a range of charts and tables which require correct interpretation.

More so, Monte Carlo simulation itself is only one piece of the risk and decision assessment pie. Decision modelling and optimisation, predictive modelling and statistical analyses should also form part of the quantitative approach to uncertainty. There is life beyond just risk simulation software, and I intend on exploring that in future blogs.

» Part 1
» Part 2

Rishi Prabhakar
Trainer/Consultant

This Thursday, 11 March 2010 at 11am ET, Dr. Jose A. Briones, SpyroTek Performance Solutions, will present a free live webcast entitled, Use of @RISK for Probabilistic Decision Analysis of a Manufacturing Forecast in an Environment of High Uncertainty

Profitability projections in a manufacturing environment are directly tied to how the sales forecast fits with the capability of the operation. When a company has a large portfolio of products with very different operational production rates, the manufacturing capacity of the plant will be significantly impacted by the product mix to be produced. This in turn will have a radical effect on the output of the plant and the allocation of the fixed cost of production. In this case we present an example where a company is trying to decide how best to balance the sales of certain families of products to maximize revenue, maintain a diverse product line, and properly price each individual product based on the impact to the manufacturing schedule and fixed cost allocation.

» Register now for this FREE live webcast
» View archived webcasts

I read an interesting article on the causes of the Global Financial Crisis by John B. Taylor. Although the topic is interesting enough already, especially for a member of a risk analysis-specialising company, something else caught my eye. I have observed in training workshops, onsite consulting and now academic papers a phenomenon regarding probabilistic modelling. Many of those using the methods don’t understand what they should actually be getting from the methodology. There is an intellectual leap from the deterministic to the probabilistic that sometimes does not get made. This limits the usefulness of Monte Carlo simulation, and the value of performing such statistical analyses.

Back to the article which spurred me to write this blog in the first place. Or rather, the graph. Yes a single graph of housing starts vs. time (and its brief description) leapt out at me. One of the lines on the graph was claimed to show model simulations of housing starts using the actual interest rate, compared to the interest rate ‘predicted’ by the Taylor Rule and a third line showing actual data.

So what’s the problem?

The problem is that simulation techniques should not be used to create a single value. The single ‘simulation’ line implies a single modelled/returned value for each time period. This is deterministic modelling. There may be a particular scenario that has been modelled, but it certainly isn’t a simulation that is being represented by that single line. Simulations produce thousands of data, observed values and their associated percentiles as well central moments (mean, variance etc.). Not just one value (sorry Value at Risk – that includes you too) that can be plotted as a single line. I would guess that if a simulation were run as I understand the term then the line in the chart was probably constructed using the simulated means. But I shouldn’t be guessing.

This is far from the only time I’ve seen simulation results reduced to a single entity. I have heard from clients in the past “the simulation gave $X” with little to no context around it, and this is supposed to both mean something to me and to their customers and help to make better decisions under uncertainty…

In the next blog I will explore this idea further and discuss the sorts of results that should be gleaned from a simulation. In particular, why narrowing simulation results down to a single number is counterproductive to healthy business practices.


Rishi Prabhakar
Trainer/Consultant

The latest Business in Britain survey from Lloyds TSB Commercial shows that the UK's commercial enterprises are regaining confidence.  The six monthly report charts the performance of 1,732 UK companies and their views on prospects for the coming year. Its most recent business confidence shows that expectations for both sales and orders have started to recover. The balance of firms anticipating an upturn in sales has climbed to 21% - from just 1% six months ago.   And hopes for orders are also looking brighter. The balance expecting order levels to rise over the coming six months has climbed to 23%, from just 6% in the last survey.

But companies planning major new business drives for 2010 would do well to follow the example of Thales UK, which uses @RISK  to enable it to assess commercial feasibility of potential new business wins. @RISK's in-depth risk analysis ensures the leading provider of mission-critical electronic information systems for aerospace, defence and security markets around the world, is fully informed when making business-critical decisions.

Thales operates in a highly competitive environment, with technologically advanced countries presenting tough opposition when it tenders for contracts. It must continually develop highly sophisticated equipment that is robust and failsafe to meet the stringent demands of its customers. Bringing products of this calibre to market is costly in terms of time and resource, so for every competitive new business opportunity, Thales must be confident that it has a reasonable chance of success.

Using Monte Carlo analysis to show all potential scenarios and the likelihood that each will occur, @RISK enables Thales to calculate the competitiveness of complex markets, measure probabilities for project costs, quantify rate of return, and even account for the effects of cumulative business, thereby providing decision-makers with the most complete picture possible.  From this risk analysis, Thales can make an informed decision on the commercial viability of the potential new business offered.

Craig Ferri
EMEA Managing Director of Risk & Decision Analysis

Two weeks ago, I attended IQPC’s (International Quality & Productivity Center) Lean Six Sigma and Process Improvement Summit in Orlando, Florida. During the past 4 years, I have watched the conference, the attendees, and their projects evolve. The IQPC did an excellent job keeping the quality of the conference at an A+ level despite wrangling with the effects of a down market and near zero travel budgets for many companies. This conference has earned it place as one of the premier Six Sigma events of the year.

With attendance numbers on par with last year (which are only slightly down from a few years ago), the major difference that I noticed was the attendees' passion. As the economy has worsened and media’s perception of Six Sigma waned, practitioners and champions are more passionate and committed now than ever. Perhaps it’s because they still have jobs and their companies understand the value of cost reduction in both their processes and product/ process development programs. They - and the companies who employ them - have every right to be excited and passionate because they are making positive changes to their organizations that will hopefully lead them to recovery and stability faster than others.
 

Many companies, large and small, represented practically every industry. Farmers Insurance and Capital One were two representatives from the insurance and banking industries. Technology and pharmaceuticals were well represented by Seagate, Motorola, Merck and Johnson & Johnson. In addition, the energy sector was well represented, as were the military, aerospace and services sectors. (If you want a complet list of companies attending, it may be available at www.sixsigmaiq.com)

The overriding message heard over and over again, was, “We need to make your Six Sigma deployments stick.” Initially, I found this to be an interesting message since it came from a group of many highly intelligent and motivated individuals who were obviously very successful in doing just that: “Making it Stick”. This message serves as a clarion call for all of us. We need to look for new tools, philosophies and approaches to make our improvement initiative better and “stickier” so that they can pass the test of time.

The highlight of every year is the awards ceremony. There were many great projects honored this year, and congratulations to the winners and everyone who submitted their projects! At the awards ceremony I had the pleasure to meet a great group from the Bahamas Telecommunications Company. They are the pioneers for Lean Six Sigma for their company. (I tried to get them to need an onsite training session in some of the Palisade tools, but have been thus far unsuccessful!) Good luck on your Six Sigma Journey, I hope to see you accepting an award next year!

This Thursday, 28 January 2010 at 11am ET, Dr. William Strauss, President of FutureMetrics, will present a free live webcast entitled, "Simulating the U.S. Economy: Where will we be in 100 years?" Sign up now to attend the webcast.

There is an assumption that drives all of our expectations for how our economy will be in the future. That assumption is one of endless economic growth. Clearly endless exponential growth is impossible. Yet that is what we base all of our expectations upon. We all agree that zero or negative economic growth is bad (just look around now at the effects of the Great Recession). But we also know logically that 2% or 4% annual growth every year leads to an exponential growth outcome that is unsustainable. 

To see where this growth imperative will take us we first have to see how we go to where we are today. This free live webcast first models the 20th century. The model is both complex and simple. The basic schematic of the model’s relationships is easy to understand. Furthermore, the core of the model is a simple production function that combines capital, labor, and the useful work derived from energy to generate the output of the economy. Complexity is contained in the solutions to the internal workings of the model. What is unique is that there are no exogenous economic variables.  Once the equations’ parameters are calibrated, setting the key outputs to “one” in 1900 results in their time paths very closely predicting the U.S. GDP and its key components from 1900 to 2006. 

The experiment in this webcast is about the future. If the model can very closely replicate the last 100 years, what does it have to say about the next 100 years? From 1900 to 2006 there are periods in which there was parameter switching. (The optimal parameters and the years for the switching were found using a constrained optimization technique.) That suggests that in the future there will also be changes. The experiment uses @RISK’s features (risk analysis software using Monte Carlo techniques) to generate new combinations of parameters for each of tens of thousands of runs of the simulation. Changes in the parameters represent potential exogenous policy choices.

The “doing what you did gets you what you got” scenario leads to a surprising and unsettling outcome. The experiments using Evolver (genetic algorithm optimization using Monte Carlo simulation) do find a path that works. Obviously if it is not “business-as-usual” that leads to a stable outcome, it is some other way. The policy choices that lead to a stable outcome suggest that the future of capitalism is not going to be what we expect it to be.

----
William Strauss is the President and founder of FutureMetrics. He brings more than thirty years of strategic planning, project management, data analysis, and modeling experience into the company’s stock of knowledge capital. Bill’s professional history includes executive positions as director, president, and senior vice president, as well as positions as senior analyst and field coordinator. He has an MBA (specializing in Finance) and a PhD (Economics).

» Register now for this FREE live webcast
» View archived webcasts

Having talked in previous posts as to why it’s important, and today how accessible it is for any size of organisation to adopt a healthy approach to risk, I’ll now take you through my top ten tips on how you can maximize your risk management programme:

1. Get buy-in
Risk management is not an optional extra. It is a business critical tool that is an asset and an integral part of the project. The company culture must be developed to embrace QRM (quantitative risk management) and DMU (decision making under uncertainty) in order that everyone understands their benefits and therefore accepts the need for them.

2. Get budget
Business tools cost money, but managing risk is an investment - not an overhead – and must be regarded as such. Allocating resource and making it a formal business process should be seen as an insurance policy.  Not only will it help organisations make better decisions that will save them money in the long term but, by identifying potential risks and adverse events, it can protect them against unexpected costs in the future.

3. Get words
As with any organisational change, it is essential that everyone is clear on the new processes. Therefore a common risk language – or 'glossary' – needs to be developed to avoid misunderstanding and to ensure a consistent approach to QRM and DMU.

4. Get numbers
Qualitative assessment is essential, but numbers are more powerful – for example the percentage chance of meeting a deadline or budget. Monte Carlo simulation random sampling provides the margin of error for a venture and is a good way to illustrate the consequences of different courses of action. Risk management experts must ensure everyone understands these figures, and accepts them.

5. Get structure
Managing risk in order to make better-informed decisions requires an appropriate organisational structure. Individuals and groups need clearly defined roles, and must then each take responsibility for their own area of expertise.

6. Get lateral
Every organisation has risks that it deals with on a daily basis and which must therefore be factored in to the decision-making model. However, no enterprise operates in isolation, so other external variables must be included. For example, even a small rise in fuel costs could have a major effect on revenues if raw materials need transporting long distances.

7. Get perspective
Political, cultural and social risk factors can be explored by involving all stakeholders.  Investing time and money in consultation and research ensures that businesses have a clear idea of the complete environment in which they operate, and therefore minimise the chances of products and services failing.

8. Get reporting
Risks, and the management of them, must be reviewed regularly – and the programme amended if necessary. This requires a regular reporting process, in which risks are clearly identified and prioritised.

9. Get with it
Being risk aware does not mean being risk averse. Businesses should guard against rigidly adhering to 'the way we've always done it' approach, instead keeping up-to-date, learning new tricks and not being afraid to be bold.  Although risky on the surface, these tactics prevent being left behind – much of the potentially uncertainty can also be removed with QRM and DMU.

And finally…

10. Get it documented
Back up the commitment to a thorough QRM and DMU programme with documentation. This validates the budget and buy-in requested at the start. And it’s good for business – organisations this thorough are guaranteed a competitive edge.

Craig Ferri
EMEA Managing Director of Risk & Decision Analysis

It’s clear that the financial crisis has exposed a number of failings in the practice of risk management. In my last post I talked about the relevance risk analysis and the disciplines of ‘quantitative risk management’ (QRM) and ‘decision making under uncertainty’ (DMU) are to all sizes of organisations, be it large or small. 

However, how accessible are these disciplines to the average size business across the globe today?

With the need to make more informed decisions more pressing by the day, thankfully QRM and DMU and now far more accessible than ever before.  Traditionally systems tended to be expensive, enterprise-based applications targeted at large companies who were prepared to spend considerable time, money and human resources.  The result was an all-singing, all-dancing product which often ended up underused due to confusion on the part of the very employees who were supposed to make it work.

Steady increases in computer processing have given the desktops of today as much power as the high-end servers of a few years ago, meaning that risk analysis and management is now an achievable goal for businesses of all sizes.  Palisades @RISK and Decision Tools Suite software are such desktop risk and decision analysis tools – working within Microsoft Excel and therefore being accessible to a large number of users.

‘Monte Carlo Simulation’, a technique originally conceived by scientists working to develop the atomic bomb as part of the Manhattan Project, is an inherent part of @RISK, a cornerstone of the Suite.  It enables users to introduce uncertainty into their previously static spreadsheets, which lets them look at things in a probabilistic, rather than a deterministic way.  In layman’s terms, this means that rather than companies and individuals making decisions based on estimates or best guesses, they can see all the potential outcomes to a venture – and how likely these scenarios are to occur.

For many companies this significantly improves the decision-making process.  Firstly it requires a change in the methodology of employees responsible for assessing risks and opportunities and secondly for the first time employees have a tool which allows them to communicate their recommendations to management or colleagues in a transparent and standardised way.  Equally, being able to look at scheduling risk in a probabilistic and quantitative sense allows for the allocation of labour and resources in a way which minimises slack and wastage whilst maximizing ROI.

So, it would seem that the new ‘risk management’ language that is starting to develop in the workplace and being taught to a new generation of managers on MBA courses should be welcomed.  With the accessibility of the technology available to assist them, we need to make sure that organisations do more than just pay lip service to QRM and DMU if they are to reap the rewards.

In my next blog I’ll be giving you the my top ten tips to adopting a health approach to risk, that will help businesses of all sizes maximize their risk management programmes.

Craig Ferri
EMEA Managing Director of Risk & Decision Analysis

I'll defer to Mr. Miller, whose blog is loaded with interesting details.   

A friend and colleague who knows I write a Six Sigma blog sent me a link to an older article on PC World, The 25 Worst Tech Products of All Time that he thought might applicable to Six Sigma.

As first blush, I thought, “What an article on PCWorld.com on the Worst Tech products would have anything to do with Six Sigma?”  The answer . . . everything! Particularly after reading the piece, the number 1 or worst product of all time (in their eyes) is American Online. I agree AOL has had its difficulties, but one has to admit the service has had staying power despite this. It’s been around for 20 years, which is a lifetime in the computer world. I don’t know if they utilized Voice of the Customer (VOC) , but they did something right since they are still around.  

The article mentions AOL had shown improvements over the previous years. This goes to show us, they had a good idea, but took many years to sort out the bugs and for them to position themselves correctly.  At the time of initial development they probably didn’t utilize Design for Six Sigma or another Critical Parameter development methodology, but it appears they may have implemented Lean Six Sigma principles to improve their “inexcusably poor customer service,” “inaccessible dial-up numbers,” and what I’ll call “flawed billing practices.” Please know I am not necessarily agreeing with the article, or being an advocate for AOL, I’m simply pointing out how the company has appeared to have improved its product and service over time.

One can only hope and assume that companies are doing a better job up front vetting their ideas, products and designs . . . with sound initiatives such as Design for Six Sigma.  If not, hopefully we won’t seem the on PC World’s next “worst of” list.


If you would like to learn more about Design for Six Sigma, May I recommend either of these two free webinars:

1. Accelerating Product Design with Simulation and Stochastic Optimization by Andy Sleeper of Successful Statistics
2. DFSS-based Design Optimization using Design of Experiments and @RISK by Jeff Slutsky Global Director of DFSS for Bausch & Lomb.
    

Wayne Winston is the newest blogging personality at the Huffington Post! His first post, “The Importance of Schedule Strength in Sports,” appeared yesterday. Wayne will focus on the interface between math and sports, with detailed explanations of statistical analysis and spreadsheet modeling, including @RISK risk analysis models. You can find a link to the Wayne Winston blog from the newly-launched HuffPost Sports.

Wayne is the John and Esther Reese Professor of Decision Sciences at Indiana University’s nationally ranked Kelly School of Business. He has won over 30 teaching awards, and written over 20 journal articles and 15 books.  Wayne has consulted for many organizations including the Dallas Mavericks, USA Diving, Cisco, Microsoft, US Army, Eli Lilly, Diamond Consulting, Tellabs and Medtronics. He has also developed online spreadsheet modeling and mathematics courses for Harvard Business School Publishing. And, Wayne is a two time Jeopardy! champion!

Wayne’s latest book, Mathletics, provides an introduction to the use of math by baseball, football, and basketball teams. He has also authored several books published by Palisade, including Financial Models Using Simulation and Optimization I, Financial Models Using Simulation and Optimization II: Investment Valuation, Options Pricing, Real Options & Product Pricing Models, and Decision Making Under Uncertainty with RISKOptimizer.


DMUU Training Team

I’ve heard of Six Sigma being used in every industry from manufacturing, banking, even baking, but now  . . . wine making?

Just the other night I found out a winery is using Six Sigma principles to ensure they are producing the highest quality wine available.
 
Yes, that’s right . . .  Six Sigma Ranch and Vineyards have combined the old-world art of wine making with the science of data driven Six Sigma principles.  Why not! Isn’t the origin of Design of Experiments from the agricultural world? That’s where (is that right?) RA Fisher introduced the concepts of replication, randomization, blocking and devise analysis of Variance to separate the sources of variation in the 1920s.

How many times have we read the reviews from a single winery, how some years are better than others, etc., and wondered why they can’t make the quality more consistent? Why not apply Six Sigma to wine making?

I think it makes perfect sense!

Six Sigma Ranch and Vineyards is applying Six Sigma principles in all stages of the process:

• Conduct extensive analyses of soil, water and climate to find the most favorable sites for our vineyards.
• Choose rootstocks that thrive best in the soil composition of a given vineyard.
• Meticulously prune vines to enhance the quality of grapes and to allow consistent ripening.
• Apply chemical and sensory analyses to pick the grapes at just the right time to produce optimal flavor in the wine.
• Listen to the voice of the customer - whether you are a sophisticated wine drinker with well-defined preferences, a social wine drinker who knows what you like and wants the security of consistency, or you just want a good place to start

The use of Six Sigma in all business process makes good sense. There is talk that Six Sigma is dead, and that people are waiting for the next big thing. The truth of the matter is no matter how you repackage the tools, these tools will be around for decades, because good decisions are based on data analysis and that should never go away.  My only hope is that they are using @RISK  to analyze their data to make even better decisions.

The next time I am in California or the local wine store, I’ll have to investigate this further.

Having successfully dodged high school physics I would not normally be sucked in by an article on quarks, but this one involved neural network computing, which I do understand pretty well.

Dr. Robert Ameo is principal of Market Modelers, LLC, with over 20 years’ experience in health care management, marketing and business development. Prior to founding Market Modelers, he served in the corporate development group at Johnson & Johnson. He is a recognized expert and innovator in the modeling and forecasting of new technology adoption and market share. Robert has extensive experience evaluating investment opportunities and their portfolio impact for mergers and acquisitions, venture investing, research development, and marketing efforts. Using his training as a psychologist and his extensive industry experience, he designs and executes targeted market and expert research experiments to quantify the defensible range of possibilities for new technology and product adoption. His forecasts are used both by start-up ventures to create a vision of their potential worth, and well-established biopharmaceutical and medical device companies to understand the true economic (uncertainty adjusted) value of their potential investments. Prior to his industry experience, Robert was VP of Clinical Operations and Utilization Management for a national managed care company. He holds a behavioral science PhD from the University of Miami.

Dr. Ameo will present a case study next week at the 2009 Palisade Conference: Risk Analysis, Applications, & Training,  21 - 22 October at the Hyatt Regency in Jersey City (10 minutes by PATH from Manhattan's Financial District).

See the abstract for his case study below, and see the full schedule for the Conference here.

Interpretive and Ethical Issues in using Monte Carlo Simulations to Support Executive Decision-Making: How to avoid giving your boss impressive, but misleading guidance

Simulations are proliferating throughout the business community powered by a troop of freshly minted MBAs armed with their requisite course on decision sciences and their student versions of @RISK.

Finance organizations are asking their analysts to “do a Monte Carlo.”  Dutifully, the analysts select a handful of “key” variables, assign triangular or Pert distributions, set iterations to 1000, push the simulate button. The laptop’s screen displays a colorful histogram and a sensitivity analysis to add to the PowerPoint.

Lo and behold, the simulation analysis supports the original scenario model showing the mean or median simulated output to be just about in the middle of the distribution. Mission accomplished. Senior leadership is assured that the model has been tested by simulating 1000 potential outcomes. Management moves forward in their pre-decided direction with confidence bolstered by a state of the art Monte Carlo analysis.

This scenario happens every day and for so many reasons it is very wrong.

Using simulations to support executive decision-making introduces ethical concerns that are not present in “most likely case” scenario modeling. In this presentation, Bob Ameo discusses the ethical responsibilities of using simulation models to inform executive decision-making. Specific recommendations are made how to appropriately conduct and present outcomes from simulation models.

Next Week: October 21-22 in NYC

Building on the success of last year’s record-breaking event, the conference will offer a wide range of software training, model building, and real-world case study sessions. Last year, the event drew over 150 practitioners and decision-makers from a broad spectrum of industries. The @RISK and DecisionTools software tracks were more popular than ever. This year, we’re expanding software training with sessions that let you walk through examples and try the tools directly. This will enable you to take some new tips back to the office. Please join us in October for a great opportunity to learn and connect with colleagues.

When performing a cost risk analysis study, one of the key results is the amount of extra monetary resources that is to be added to the project cost baseline to guarantee that the budget is not exceeded at a certain confidence level. Good project risk management strategies must take this into account.

After defining the uncertain variables and risk events that affect the cost performance of the project, we can run a Monte Carlo simulation with @RISK to find out what the range of the total project cost is.  Simulation results can help us to explain the risk exposure that we have in the total cost of the project. The most popular statistics are the mean (average cost), the most likely cost, and the 10th and 90th percentiles.




To determine the contingency to be allocated to the project, we need to define what confidence level we would like to achieve: The higher the contingency level, the larger amount of contingency needed. For example, in the figure above, we are reporting the total cost of the project. Here we can observe that we are showing the 85th percentile that corresponds to a total cost of $7.8M (right delimiter).  We can say that there is only a 15% chance that we will exceed $7.8M, or alternatively, we have an 85% chance that the total cost will be less than or equal to $7.8M.  In the same figure we can also see that the 90th percentile of the total project cost is $8.02M.  We can say then that in order to increase our confidence level from 85% to 90%, we will need to add $220,000 to the total cost.

The calculation of the contingency is then accomplished by using the base cost estimate (BE) before the risk analysis was implemented, and the expected cost (EC) of the simulated results.

Some practitioners separate the contingency into two components: engineering allowance, and management contingency.

Engineering allowance (EA) is the difference between the expected cost and the base estimate:

EA = EC – BE

Management contingency (MC) is calculated using the difference between the cost at certain confidence lever (Cp) and the base estimate:

MC = Cp – EC

In our example, our BE = $6.5M; therefore, engineering allowance EA = EC – BE = 6.86M – 6.5M = $0.36M. 

For the calculation of management contingency, we use a confidence level of 85% so Cp(85%) = $7.8M; therefore, MC = Cp – EC = 7.80M – 6.86M = $0.94M.

In many situations, the suggested contingency might be excessive, so the need for a mitigation study is necessary. We can use the sensitivity analysis tool in @RISK to detect the key drivers affecting our total cost. This is valuable information so that we can concentrate our efforts in reducing the impact of risk events and uncertainties to the total cost. Below, we see a tornado graph with the most important drivers. The analyst will then explore the appropriate mitigation strategies and assess their implementation cost. A second simulation can be run to assess the effectiveness of the proposed mitigation plan, and compare the pre-mitigated and post-mitigated cost distributions.




In following blog posts, I will explain how to distribute the assessed contingency to cost elements and identified risk events in project risk management models.

Javier Ordóñez, Ph.D
Director of Custom Solutions