Monte Carlo Techniques

Abbott suggests a compromise: streamline your plot by masking (Abbott says "removing") fields that contribute to the robustness of the analysis but involve statistical twists and turns that are distracting to decision makers who may not be fascinated with technique and just want to see how the story turns out. This, he explains, allows you to work from a model both you and the decision makers can believe in.

Your thoughts? 

With the New York State primaries coming up September 14 and the general election on November 2, I predict that as soon as summer turns the corner into September, we'll start hearing lots and lots about polls that predict election outcomes.  To find out if there was any early discussion of polls, polling, and outcomes, I returned to my favorite election forecast site from the 2008 presidential elections, FiveThirtyEight: Politics Done Right.

No matter how good your model, it's still You Play, You Pay.  And Rowland's disclaimer echoed a comment an influential racing veterinarian made to me: "Never invest in something that eats while you sleep."     

Just after I posted my last blog questioning a recent Investopedia column in the San Francisco Chronicle, I had a congenial note from the author of that column, David Harper.  His column compared Monte Carlo Simulation with two other methods of calculating Value-at-Risk, and I was concerned that its view of risk and risk analysis techniques was overly simplified. David   was surprised to discover that column had just appeared because he wrote it five years ago!

The five-year lag explains a lot--Monte Carlo simulation was not nearly so widely adopted or carried about by so many software tools as it is today--and I should have suspected the article was a vintage piece before I started carping.

So I happily retract my concerns to introduce to you David Harper, CPA and certified Financial Risk Manager.  In response to my comment about the attitudes and techniques that led to last year's collapse of the financial markets, David says that, now that the black swan has flown, "the crisis should implicate both HistoricalSim VaR and parametric VaR (at least multivariate normal!) and point toward Monte Carlo Sim. I've been thinking for a while that all of this [I think he means lack of accuracy in specifying risk] should really boost Monte Carlo."

Investment commentary is only one of David's activities.  He is the founder of Bionic Turtle, a business devoted to e-learning about financial risk and preparation for the certification exam for financial risk managers. This is a worthy enterprise--I was relieved to discover that there are hoops financial risk managers have to got through to be called that--and for anyone who would like to know more about quantitative techniques for risk analysis, its website is worth prowling. 

Thank you, David, for setting me straight.  

Example Model: SENSIM.XLS

Sensitivity analysis in @RISK (risk analysis software using Monte Carlo simulation) lets you see the impact of uncertain risk analysis model parameters on your results. But what if some of the uncertain model parameters are under your control? In this case the value a variable will take is not random, but can be set by you. For example, you might need to choose between some possible prices you could charge, different possible raw materials you could use or from a set of possible bids or bets. To properly analyze your model, you need to run a simulation at each possible value for the "user-controlled" variables and compare the results. A Sensitivity Simulation in @RISK allows you to quickly and easily do this - offering a powerful analysis technique for selecting between available alternatives.

In @RISK any number of simulations can be included in a single Sensitivity Simulation. The RiskSimtable function is used to enter lists of values, which will be used in the individual simulations, into your worksheet cells and formulas. @RISK will automatically process and display the results from each of the individual simulations together, allowing easy comparison.

» Click here to see how to run a Sensitivity Simulation
» Click here to download the example file SENSIM.XLS

At the 2010 Palisade Risk Conference in London, John Zhao of Statoil used a mock cost estimate contingency model to demonstrate how @RISK simulation functions can yield a more realistic project contingency through integrated qualitative risk assessment and quantitative risk analysis.

While future oil prices may be hard to predict due to low manageability, it is absolutely possible to scientifically forecast the sizes of risks that companies are willing to take, and such risks may include the probabilistic volumes of newly discovered reserves, the probability of meeting a project development schedule, chances of project cost overruns, and the likelihood of eroding entire project profitability. To achieve these goals, @RISK has lent a helping hand to business analysts for easier operation of complicated mathematical modelling.

Statoil, an international oil company, takes risk management seriously and has applied Monte Carlo simulation techniques in core and support businesses using @RISK. Such applications not only include the solo use of individual applications, but integrated combinations from drilling, reserve estimation, and well completion to cost and schedule controls at project execution. Besides the widespread uses of the software, Zhao discussed a specific application of @RISK to convincingly simulate required capital project contingency  in detail.

A simplistic line-item ranging exercise using @RISK Monte Carlo simulation is no longer adequate to derive large capital project contingency, as empirical data confirmed that many disastrous cost overrunning projects were lack of contingency to cover the covert risks. In order to show management a complete risk picture on a project, both systemic risks (which empirical history has indicated a likelihood of occurring), and specific risks (which have discrete probabilistic characteristics), should be included in the overall project risk analysis. Therefore the combination of continuous PDF for project cost estimates, and discrete PDF for project risk registers, may prevail and provide management with a more convincing project cost contingency.

John Zhao is Quality and Risk Manager at StatoilHydro Canada Limited. He has 22 years project management experience in the petrochemical industry. He has authored many papers and made numerous presentations worldwide on the subject of risk and contingency management. In the past 10 years, John has developed his expertise in cost engineering and risk analysis for large downstream and oilsands upstream projects across Canada. His extensive knowledge in construction project qualitative risk assessment process has made him an expert on the subject in North America; his proprietary Monte Carlo model using @RISK is a popular tool for project contingency and escalation simulation. The quantitative model that John has built has integrated @RISK with PrecisionTree to help corporations conduct risk-based strategic decision-making.

» View the complete abstract and PDF presentation of "Put More Science into Cost Risk Analysis"
» Read Zhao's whitepaper, "Put More Science into Quantitative Risk Analysis"

At the 2010 Palisade Risk Conference in London, David Inbar of Minet Technologies presented a talk on supply chain risk management.

Supply chain risk management is an emerging field which has been growing significantly in importance because of modern management concepts such as lean, globalization and outsourcing. The mutual dependencies and close collaboration in modern supply chains create unique risks and challenges. Supply chain risk management is an economic process and choosing the elements and amount of risk mitigations should be based on economic measures.

Inbar's talk gave an overview of the concepts and process of supply chain risk management, and demonstrated how using Monte Carlo simulation techniques with @RISK risk analysis software adds value to the decision making under uncertainty processes and enables managers to purchase the most cost effective mitigations. Says Inbar, "An organization with the right risk management process can assure peace of mind to customers and supply chain partners."

David Inbar is the founder and managing director of Minet Technologies, a provider of professional services and technologies in supply chain and purchasing. Minet is active in the interfaces between business, processes and technologies in the world of supply chain and purchasing, creating methodologies and delivering projects and solutions.

» View a PDF of the presentation here
» Abstracts and presentations from the 2010 Palisade Risk Conference in London

In my last blog I introduced the idea of a customised risk analysis solution to problems commonly faced in project risk management, especially cost estimation. Of course this idea is not uniquely applicable to project costs, but this paradigm is the simplest to explore, and that’s what I’m about to do.

Picture a risk register in a worksheet that has been created at a macro level to encapsulate most (all?) of the risks your projects may face. For any given project only a subset of these will be relevant – what is the best way to get these risks into a risk model on the next worksheet? By pressing a button of course! It is almost trivial to write code that picks up all selected risks and places them and the relevant data fields in the model worksheet. Sure beats manually copying and pasting individual line items and the transcribing errors that follow.

The next problem is utilising the workshopped parameters (likelihood of event, three-point estimates for severity etc.) in a logical way to be referenced by appropriate @RISK functions. Once a model structure has been agreed upon a macro button can place @RISK distributions where they ought to go, either logically due to the paradigm (using RiskBinomial, for example) or via a drop-down selection for dollar impact (RiskPert or RiskGamma, say). My clients have been especially thankful when I limit their choice of distribution and provide a simple flow-chart to follow to make this very decision. Reducing the propensity for arguments in risk workshops is worth its weight in gold; if we can assume that reducing this risk ‘weighs’ plenty!

Similarly one or two instances of the simulation settings are likely to satisfy all requirements, so these too can be activated by macro buttons. In this way a user can’t run a ‘poor’ simulation thus creating spurious results. The simulation output that is required can be placed into a report template attached to the model template and generated using yet another simply-labelled macro button. In this way there will be consistent reporting across the organisation allowing decision makers to become familiar and comfortable with simulation results they might otherwise ignore or be unaware of.

A risk model created by this process may not be the theoretically optimal one, but it will be valid and in context with its intended use. It will certainly be easy to use! The results will be consistent and should satisfy management’s desires as well as regulatory requirements.
The project cost estimation is but one example, and the above possibilities are far from the only ones imaginable. Additional complexity or alternate needs would be just as easily met simply with different code essentially without any practical limits. You don’t need to be an expert in Monte Carlo techniques and software to run robust, credible risk analyses. All you need is a risk analysis consultant who macro-controls the cumbersome and probabilistic elements, some appropriate simulation options and reporting procedures. Ask for me by name!

» Robust Risk Analysis for the Time/Expertise Poor – Part 1

Rishi Prabhakar
Trainer/Consultant

I have recently spoken to several clients whom have all came to the same conclusion about the risk analysis solution they think is most appropriate. They don’t want to do it, and I have no problem with that!

Of course that’s not precisely true. The benefits of Monte Carlo techniques in risk analysis are quite well understood and there is plenty of buy-in from businesses in the Australasian region. The trouble these businesses face (particularly in the realm of project cost estimation) is that the specific process of quantifying their risks for stochastic analysis and the ensuing simulation is not well understood and the means to ameliorate this appears to be beyond their reach. The modelling and simulation components of the project risk management process are not given adequate resources to be performed well, and certainly not to the extent that they provide the most useful information.

It is the case that many companies do not employ dedicated quantitative analysts. This means they have to rely upon some (maybe one) person in the team who has a non-zero quantity of experience and possibly training with risk simulation software to create a valid and credible stochastic model. This person is also not likely to be given enough time to do said task, thus the model inevitably suffers. It is my experience that most models – and all project cost estimation models – can be improved or actually need to be fixed.

So the corporate mind is willing, but the flesh is weak. How can this be addressed? No amount of additional training will suddenly allow you to overcome your time and resource constraints. Perhaps you can’t get the budget for training anyway or don’t want to master risk analysis software when it’s not really core to your role? The solution is one that I personally endorse (and provide!) as a risk analysis consultant – custom Excel programming.

VBA for Excel is a fairly simple language to learn, yet very powerful tool for automating repetitive or sometimes complex spreadsheet tasks. A customised solution involves writing VBA code to perform the tasks we’d rather not do ourselves in the risk analysis model. The “we” here refers to companies that find themselves in the situations previously described whereby they are incapable of creating and operating these models, not necessarily though any fault of their own. In my next blog I’ll examine some modelling problems/requirements and how they might be dealt with effectively using customisation.

Rishi Prabhakar
Trainer/Consultant

Modeling from empirical data takes observed information and attempts to replicate that information in a set of calculations. There are a number of relationships to account for when incorporating those data in a model. These relationships include dependencies and/or correlations. Correlations are often omitted for a variety of reasons, which can lead to critical errors in your results. Some knowledge of the situation leads to a more credible representation of the relationships in the data. Added knowledge, perhaps from subject matter experts, or other sources, aids the refinement of the conclusions one can draw from the data. Whether the correlations are direct or aggregate, involving simple mathematics or greater complexities, ultimately the model is likely to be used in some form of analysis for projecting future outcomes. The knowledge brought to the model and the analysis with embedded correlations improves knowledge about inherent uncertainty in a given problem.

Correlation is a principal relational element  which describes relationships between variables in datasets. There may be general tendencies and patterns which drive the input risks to move together or differently from each other. It is these relationships between variables which need to be expressed in a model to bolster its usefulness, which is accomplished with correlation. It is important to remember there may be observed correlation between variables but it is not necessarily a causal relationship; it may be only a general tendency of paired behavior.

One significant aspect to note: positive correlations appear to increase uncertainty. Wait, you say, how is that possible? Knowledge is supposed to reduce uncertainty. Doesn’t knowing counteract unknowing? Think about it for a moment. In effect, the correlations included in the model reduce the uncertainty about reality while increasing the range of predicted values, adding uncertainty. What may seem illogical on the outset really is quite logical. If two (or more) risks are positively correlated, their aggregation will produce a larger range as a consequence of Monte Carlo sampling. In fact, failing to account for correlations that really are there reduces the validity of the analysis.

Correlations are easily incorporated in models set up for Monte Carlo simulation. MCS, as a technique, generates many ‘random’ samples allowing the modeler to study a variety of scenarios and their impact on decisions. A correlation matrix defines the sampling relationship between any pair of input variables in the model. Using a tool such as Palisade’s @RISK facilitates matrix construction. Once the correlations are in place, running the MCS will produce results and scenarios that are more credible. We want decisions to be based on the best information available and the correlations lend a hand to the knowledge we already incorporate into the process.

Thompson Terry
Senior Training Consultant

So.  Is it time to shut down your Monte Carlo software?  I don't think so. . . .   

Something has troubled me for some time regarding the choices being made in risk land. I train and work with many clients whom have adopted Monte Carlo simulation techniques (via @RISK for Excel) into the day-to-day running of their businesses. By doing so they (hopefully) now have a good understanding of the exposure they are facing be it in project cost estimation, discounted cash flow analysis or, well, anything really. But this is only one facet of risk and decision assessment, specifically dealing with the descriptive statistical output from a simulation. What of the decision evaluation component? Why aren’t more of my customers analysing the decisions they make, or better yet actually optimising them? I have a few ideas why.

If you’re in business you have to make decisions. Big ones, little ones, yes/no, multiple state and continuous value decisions. Decisions that impact other decisions in simple or complex dependency structures. But are you making the best decisions possible? I’m sure important decisions aren’t being made completely randomly (I hope!) but I see many companies who rely completely upon qualitative techniques for their decision making (experience, gut feel, etc.) which of course means optimality is no more than a hoped for outcome rather than something that is actually being worked towards.
Firstly the decision model must be identified and then quantified, and this can be a difficult task. There is a level of modelling aptitude necessary for effective modelling that goes beyond merely knowing Excel and its functions, and into the construction of logical mathematical descriptions of possibly complicated processes. Relevant decisions need to be identified and the impact of those decisions combined into a formula that can be mathematically optimised. A critical component to all this is the knowledge that spreadsheet models can actually be optimised, and that in cases where Excel’s Solver fails there are Palisade products (Evolver and RISKOptimzer) that can perform optimisations under virtually any circumstance.

I too used to focus on Monte Carlo simulation rather than decision evaluation, and this was mainly a product of the clients I was dealing with almost exclusively when I first worked for Palisade. In my next blog I’ll tell you why that changed and also get a little more into the nuts and bolts of optimisation.

Rishi Prabhakar
Trainer/Consultant

I read an interesting article on the causes of the Global Financial Crisis by John B. Taylor. Although the topic is interesting enough already, especially for a member of a risk analysis-specialising company, something else caught my eye. I have observed in training workshops, onsite consulting and now academic papers a phenomenon regarding probabilistic modelling. Many of those using the methods don’t understand what they should actually be getting from the methodology. There is an intellectual leap from the deterministic to the probabilistic that sometimes does not get made. This limits the usefulness of Monte Carlo simulation, and the value of performing such statistical analyses.

Back to the article which spurred me to write this blog in the first place. Or rather, the graph. Yes a single graph of housing starts vs. time (and its brief description) leapt out at me. One of the lines on the graph was claimed to show model simulations of housing starts using the actual interest rate, compared to the interest rate ‘predicted’ by the Taylor Rule and a third line showing actual data.

So what’s the problem?

The problem is that simulation techniques should not be used to create a single value. The single ‘simulation’ line implies a single modelled/returned value for each time period. This is deterministic modelling. There may be a particular scenario that has been modelled, but it certainly isn’t a simulation that is being represented by that single line. Simulations produce thousands of data, observed values and their associated percentiles as well central moments (mean, variance etc.). Not just one value (sorry Value at Risk – that includes you too) that can be plotted as a single line. I would guess that if a simulation were run as I understand the term then the line in the chart was probably constructed using the simulated means. But I shouldn’t be guessing.

This is far from the only time I’ve seen simulation results reduced to a single entity. I have heard from clients in the past “the simulation gave $X” with little to no context around it, and this is supposed to both mean something to me and to their customers and help to make better decisions under uncertainty…

In the next blog I will explore this idea further and discuss the sorts of results that should be gleaned from a simulation. In particular, why narrowing simulation results down to a single number is counterproductive to healthy business practices.


Rishi Prabhakar
Trainer/Consultant

There is an assumption that drives all of our expectations for how our economy will be in the future.  That assumption is one of endless economic growth. Clearly endless exponential growth is impossible. Yet that is what we base all of our expectations upon. We all agree that zero or negative economic growth is bad (just look around now at the effects of the Great Recession). But we also know logically that 2% or 4% annual growth every year leads to an exponential growth outcome that is unsustainable. 

In this free webcast, Dr. William Strauss models the next 100 years, based on the last century's data. The experiment in this webcast is about the future. If the model can very closely replicate the last 100 years, what does it have to say about the next 100 years? The experiment uses @RISK’s risk analysis and Monte Carlo techniques to generate new combinations of parameters for each of tens of thousands of runs of the simulation. Changes in the parameters represent potential exogenous policy choices.

The “doing what you did gets you what you got” scenario leads to a surprising and unsettling outcome. The experiments using Evolver (genetic algorithm optimization software) do find a path that works. Obviously if it is not “business-as-usual” that leads to a stable outcome, it is some other way. The policy choices that lead to a stable outcome suggest that the future of capitalism is not going to be what we expect it to be.

Palisade is pleased to host this presentation from Dr. William Strauss.

William Strauss is the President and founder of FutureMetrics. He brings more than thirty years of strategic planning, project management, data analysis, and modeling experience into the company’s stock of knowledge capital. Bill’s professional history includes executive positions as director, president, and senior vice president, as well as positions as senior analyst and field coordinator. He has an MBA (specializing in Finance) and a PhD (Economics). Read more of Dr. Strauss' bio here.

» Complete abstract of "Simulating the U.S. Economy: Where will we be in 100 years?" 
» Register now (FREE)  
» View archived webcasts

This Thursday, 28 January 2010 at 11am ET, Dr. William Strauss, President of FutureMetrics, will present a free live webcast entitled, "Simulating the U.S. Economy: Where will we be in 100 years?" Sign up now to attend the webcast.

There is an assumption that drives all of our expectations for how our economy will be in the future. That assumption is one of endless economic growth. Clearly endless exponential growth is impossible. Yet that is what we base all of our expectations upon. We all agree that zero or negative economic growth is bad (just look around now at the effects of the Great Recession). But we also know logically that 2% or 4% annual growth every year leads to an exponential growth outcome that is unsustainable. 

To see where this growth imperative will take us we first have to see how we go to where we are today. This free live webcast first models the 20th century. The model is both complex and simple. The basic schematic of the model’s relationships is easy to understand. Furthermore, the core of the model is a simple production function that combines capital, labor, and the useful work derived from energy to generate the output of the economy. Complexity is contained in the solutions to the internal workings of the model. What is unique is that there are no exogenous economic variables.  Once the equations’ parameters are calibrated, setting the key outputs to “one” in 1900 results in their time paths very closely predicting the U.S. GDP and its key components from 1900 to 2006. 

The experiment in this webcast is about the future. If the model can very closely replicate the last 100 years, what does it have to say about the next 100 years? From 1900 to 2006 there are periods in which there was parameter switching. (The optimal parameters and the years for the switching were found using a constrained optimization technique.) That suggests that in the future there will also be changes. The experiment uses @RISK’s features (risk analysis software using Monte Carlo techniques) to generate new combinations of parameters for each of tens of thousands of runs of the simulation. Changes in the parameters represent potential exogenous policy choices.

The “doing what you did gets you what you got” scenario leads to a surprising and unsettling outcome. The experiments using Evolver (genetic algorithm optimization using Monte Carlo simulation) do find a path that works. Obviously if it is not “business-as-usual” that leads to a stable outcome, it is some other way. The policy choices that lead to a stable outcome suggest that the future of capitalism is not going to be what we expect it to be.

----
William Strauss is the President and founder of FutureMetrics. He brings more than thirty years of strategic planning, project management, data analysis, and modeling experience into the company’s stock of knowledge capital. Bill’s professional history includes executive positions as director, president, and senior vice president, as well as positions as senior analyst and field coordinator. He has an MBA (specializing in Finance) and a PhD (Economics).

» Register now for this FREE live webcast
» View archived webcasts

Today there is a heightened appetite for risk management due to global economic circumstances. But risk management has always been an intrinsic aspect of business to a higher or lesser degree. However, in the current technology-led business environment, the use of software to effectively manage risk makes logical sense. It provides a level of sophistication that the traditional processes simply cannot offer. Let me explain why.

Risk management essentially involves three stages – identification, quantification, and the on-going management of risks. In reality, these stages are not completely distinct from each other, with each stage influencing and informing the others. For example, an initial quantification of risks may lead to the conclusion that some of the identified risks are in fact not serious enough to warrant further consideration, or that the original description of the risk was not sufficiently precise for meaningful risk management measures to be put in place.

Each of these stages can benefit from the use of supporting risk modeling software. For instance, Microsoft Excel can be used to create a risk register, i.e. a database that records the risks identified, the assessment of the likelihood and impact of each of these risks, the mitigating actions that have been planned, and the assignment of responsibilities for these actions. However, there are many other software tools available, each designed for a specific purpose and focus. To illustrate, enterprise-wide risk management software focuses on the creation of integrated and holistic risk management systems, whereas Monte Carlo simulation and decision tree software place their emphasis on enhancing the quantitative analysis of risks.

The selection of the appropriate risk analysis software should involve very careful thought. The right decision can lead to a very effective implementation, whereas the wrong decision may result in a large amount of wasted investment.

There are some key considerations to bear in mind when selecting the risk modeling software. Choosing software based on how many staff will genuinely be required for the day-to-day risk management process is crucial. It is easy to select software based on the ideal situation that there will be a wide staff involvement in the risk management process. In reality, this may not be possible, potentially resulting in a cumbersome and inflexible solution being chosen over a more stand-alone and flexible application.

Similarly, knowing the level of risk quantification required is important. In fact, best practice risk management now involves the use of quantitative techniques, often using Monte Carlo simulation. When correctly conducted, the process of quantifying risks is rigorous and structured, can expose hidden or biased assumptions, as well as provide a more solid rationale upon which to base the major decisions.

Finally, determining the extent of on-going risk management needed for your business can assist with software selection. 

Needless to say, any software application will be most successful when used by appropriately trained and motivated staff, and when used as a supporting tool within an overall risk management process. Software is not a replacement for process.

Craig Ferri
EMEA Managing Director of Risk & Decision Analysis

It’s clear that the financial crisis has exposed a number of failings in the practice of risk management. In my last post I talked about the relevance risk analysis and the disciplines of ‘quantitative risk management’ (QRM) and ‘decision making under uncertainty’ (DMU) are to all sizes of organisations, be it large or small. 

However, how accessible are these disciplines to the average size business across the globe today?

With the need to make more informed decisions more pressing by the day, thankfully QRM and DMU and now far more accessible than ever before.  Traditionally systems tended to be expensive, enterprise-based applications targeted at large companies who were prepared to spend considerable time, money and human resources.  The result was an all-singing, all-dancing product which often ended up underused due to confusion on the part of the very employees who were supposed to make it work.

Steady increases in computer processing have given the desktops of today as much power as the high-end servers of a few years ago, meaning that risk analysis and management is now an achievable goal for businesses of all sizes.  Palisades @RISK and Decision Tools Suite software are such desktop risk and decision analysis tools – working within Microsoft Excel and therefore being accessible to a large number of users.

‘Monte Carlo Simulation’, a technique originally conceived by scientists working to develop the atomic bomb as part of the Manhattan Project, is an inherent part of @RISK, a cornerstone of the Suite.  It enables users to introduce uncertainty into their previously static spreadsheets, which lets them look at things in a probabilistic, rather than a deterministic way.  In layman’s terms, this means that rather than companies and individuals making decisions based on estimates or best guesses, they can see all the potential outcomes to a venture – and how likely these scenarios are to occur.

For many companies this significantly improves the decision-making process.  Firstly it requires a change in the methodology of employees responsible for assessing risks and opportunities and secondly for the first time employees have a tool which allows them to communicate their recommendations to management or colleagues in a transparent and standardised way.  Equally, being able to look at scheduling risk in a probabilistic and quantitative sense allows for the allocation of labour and resources in a way which minimises slack and wastage whilst maximizing ROI.

So, it would seem that the new ‘risk management’ language that is starting to develop in the workplace and being taught to a new generation of managers on MBA courses should be welcomed.  With the accessibility of the technology available to assist them, we need to make sure that organisations do more than just pay lip service to QRM and DMU if they are to reap the rewards.

In my next blog I’ll be giving you the my top ten tips to adopting a health approach to risk, that will help businesses of all sizes maximize their risk management programmes.

Craig Ferri
EMEA Managing Director of Risk & Decision Analysis