Credit Risk Analysis

Abbott suggests a compromise: streamline your plot by masking (Abbott says "removing") fields that contribute to the robustness of the analysis but involve statistical twists and turns that are distracting to decision makers who may not be fascinated with technique and just want to see how the story turns out. This, he explains, allows you to work from a model both you and the decision makers can believe in.

Your thoughts? 

Where I left off last time was lamenting the use of Monte Carlo simulation to create a single value (statistic etc.) from a model. It might still not be clear why this is anathema to me, so here goes.

A simulation is not a number. It’s not one possible (future) outcome – that’s a scenario. Monte Carlo simulation is a methodology for understanding one’s exposure to outcomes not situated close to the central tendency of the process/project in question. Note the plural “outcomes”. Risk analysis, when done properly, should let you know essentially all possible outcomes and how likely they are for your model. Output from a simulation can include a plot of means (over time), or P5s, or P95s, or the mean ± one standard deviation or any number of statistics. But that’s not plotting a simulation! Let’s not give a minimalist graph too much credit.

Such statements also perpetuate the idea that simulation is only used for creating means (or other centrally tending statistics) and ignores the wealth of information available. Risk simulation software exists to help you do risk analysis which must include not only several statistics but also sensitivity information. It is all too easy to turn a risk assessment into a hunt for a regularly asked for percentile (such as the P90) and there ends the task. I see this a lot, especially in project cost estimation where the pressure both from management and regulatory bodies is to accurately estimate some large percentile. Once found there is usually scant further risk analysis.

Nothing good ensues. When risk analyses are run “to get ‘the’ number” they become simply another box to tick in a process and ultimately any benefits (perceived or actual) will be forgotten and lost to the ages. The notion of context is also lost. No single number by itself really means anything, or at least shouldn’t mean anything to a decision maker. I have often heard phrases like “the model returned/gave $1.2m” followed by an audience nodding in agreement. Huh? Which statistic are you talking about there, and how about reporting a few other numbers around it to place that $1.2m somewhere meaningful?

In the next installment I will look further into this issue of context and hopefully prove the necessity of an holistic approach to understanding and reporting simulation results.

» Part 1

Rishi Prabhakar
Trainer/Consultant

It's a deadly game of chance, the perfect venue for Monte Carlo to meet Monte Carlo. 

The recent Wall Street Journal article “Odds-On Imperfection: Monte Carlo Simulation” asserts that Monte Carlo simulation did not predict the market crash, and cites a chorus of critics calling for a fix to the technique. The article equates the technique of Monte Carlo simulation with the models that are using it – two very different things. For instance, the article states, “These models were supposed to help quantify and manage the risks of mortgage-backed securities, credit-default swaps and other complex instruments. But given the events of the past couple of years, it appears that the models often gave big institutions, as well as small investors, a false sense of security.”

This is true – the models for decision making under uncertainty gave a false sense of security. But that’s because the assumptions underlying the risk analysis models were flawed, not because the technique of Monte Carlo simulation was problematic. Monte Carlo simulation is simply a mathematical technique that recalculates many different possible scenarios – but only within boundaries set by the user.  You can’t change the underlying math behind these “what-if” calculations.

The article comes close to making this distinction in one sentence: “Critics emphasize that the problem isn't Monte Carlo itself, but the assumptions that go into it.”  It then goes on to describe efforts by firms to include “fatter tail” distributions that more accurately reflect the probability of extreme events occurring as an effort to improve Monte Carlo simulation.  Tools like @RISK (risk analysis software add-in for Microsoft Excel) allow complete control over the definition of many dozens of distribution types, enabling users to create as fat a tail as they want. While these efforts make sense, it should be made clear that these are changes to underlying model assumptions, not changes to Monte Carlo simulation itself. To equate Monte Carlo simulation as a technique with the probability distributions people decide to use is to equate a carpenter’s choice of nails with his hammer.

Finally, the article cites the need to run tens or hundreds of thousands of scenarios, instead of just 100 or 1000.  This too is user-defined, and tools like @RISK can run as many scenarios as desired.

Randy Heffernan
Vice President

In assigning blame for what they are now calling "the credit crunch," the news media have been pointing vaguely in the direction of risk assessment and the models produced by Monte Carlo simulation.  But with the exception of Joe Nocera's excellent piece focusing on value-at-risk in the New York Times, I had not seen a clear explanation of the factors feeding into the exponential decline of the credit markets until I came across a policy paper from the Association of Chartered Certified Accountants, "Climbing out of the Credit Crunch."

This paper provides an excellent, plain-English account of the interplay of the many factors that brought the current turmoil in the financial sector--a number of these factors the ACCA identifies are psychological and attitudinal.  Its discussion of risk identification and management focuses not on risk analysis models but on the underlying assumptions--a common "garbage in, garbage out" observation--and a passive, unquestioning reliance on these models.  This leads the ACCA to one of what it identifies as a major risk management failure: "a very clear disconnect between incentives to senior staff" and [highly sophisticated] risk management functions."

The point is that risk management is so crucial to financial performance that executives who keep a close, critical eye on the tools and techniques they are using to assess risk should be rewarded for that vigilance. 

Warren Buffet did not have a good year.  This fact about 2008 was detailed last week in a New York Times article and in a letter from Mr. Buffet to stockholders of his Berkshire Hathaway company.  This letter offers plenty of plain-spoken criticism of the risk assessment failures in the financial sector to go around--'beware of geeks bearing formulas." 

But he reserves his harshest criticisms for derivatives--"weapons of mass destruction.".  He not only cites the hazards of using risk analysis models to value these complexly structured investment products, but he points out something I haven't seen mentioned before: derivatives create a "web of mutual dependence" among financial institutions that lingers for years.

About this web of dependence, he says, “Participants seeking to dodge troubles face the same problem as someone seeking to avoid venereal disease,” he wrote. “It’s not just whom you sleep with, but also whom they are sleeping with.”

I recommend Warren Buffet's letter if only for its Olympian view of the exponential decline in the credit markets and his list of the intriguing entertainments he has planned for his shareholders at their annual meeting this May.  His bad year makes for some good reading.

The recent credit crisis has brought into focus some of the difficulties in estimating and calibrating risk analysis models in which events of low probability are used.  For example, suppose a AAA-rated security is deemed to have a 1% chance of default in a particular year.  How good is that 1% estimate?

More generally, suppose historic data has 100 trials in which an event has occurred once, or perhaps 1000 trials in which an event has occurred 10 times.  The “maximum likelihood estimator” (corresponding to human intuition) assigns a 1% chance for the “true but unknown” probability of such an event. Intuitively there is however a range of possible probabilities.  For example, for the case of 10 occurrences from 100 trials, standard probability theory shows that with a true probability of 10% probability this outcome would be observed about 13.2% of the time, whereas for a 9% probability it would be observed about 12.4% of the time (so that the 10% estimate is indeed slightly more likely than the 9% estimate).

In fact is well known that the uncertainty distribution for the probability of a binomial process given some observations from a certain number of trials is represented by the Beta distribution (the assumption of a binomial process will be adequate for practical purposes when dealing with low probability events, rather than say a Poisson).  A more detailed explanation of the Beta distribution and a spreadsheet example is in Chapter 4 of my book Financial Modelling in Practice, with many examples of financial risk analysis applications.

The screenshot shows a table of the most likely estimate (1%), as well as the frequency in which the true probability (using a beta distribution) is above that estimate.  The various table entries show an increasing number of trials, with the number of observations always equal to 1% of that. The graph shows the Beta distribution for various cases.

The key points about these results are: 1. The beta distribution is skewed, but becomes more symmetric as the number of trials increases 2. The range (standard deviation) of the beta distribution narrows as the number of trials increases, so that we become more confident that the estimate is closer to the true figure 3. The total number of trials needs to be around 1600 (16 occurrences) for – in about 95% of cases - the true probability to lie within a +/-50% band around the most likely estimate (i.e. in the range 0.5% to 1.5%).



Note also that the RiskTheo statistics functions in @RISK enable these calculations to be performed directly in Excel (e.g. RiskTheoStdDev to calculate the standard deviation of the beta distribution, and so on).


Dr. Michael Rees
Director of Training and Consulting

With New Year's now in the rear view mirror, I've been speculating on what kind of resolutions the folks on Wall Street made this week.  This year's dismal performances in the markets leave a lot of room for good intentions, and I suspect one common theme of these resolutions was "better tools."   Last quarter, in the laying of blame for the credit failures that caused the financial system to buckle, there were many aspersions cast about the models used by credit rating agencies for risk assessment and decision evaluation in investing, and there were even some fingers pointing directly at Monte Carlo software and the other computational tools for risk analysis.  

Sharper tools may help protect future investors making decisions under the usual levels of uncertainty, but the fact is that shiny new tools are easier to develop than flawless judgment.   And the judgment of the model creator is a huge factor in the reliability of the model: "garbage in, garbage out," as is famously said about any kind of computer output.  

The judgment of the modeler is particularly crucial when it comes to the handling of probabilities. Therefore, RESOLVED: my first entries in 2009 will be devoted to the role and specification of probability.

Many commentators on the current financial woes in the U.S. have blamed the credit crunch on the "CDO"--collateralized debt obligation.  CDOs are an unregulated type of credit product, asset-backed securities constructed from portfolios of fixed-income assets. They come in many shapes and sizes, and they are rated in a similar way as bonds are rated. These are not simple products, and therefore the risks associated with buying and selling CDOs are not easy to quantify.  It's a tricky business, like option valuation.

As the current financial turmoil has made painfully clear, either the risk analysis done by credit rating agencies and by institutions buying and holding CDOs was not adequate or these risk assessments were optimistically ignored.  Furthermore, some pundits have suggested that because CDOs are not sold on the open market, they are not priced according to their risks--in other words, they were too easy to acquire.

How should we evaluate investment risk in a package of many debts, each of which would be assigned a different value-at-risk at any particular point in time?  We have a lot of slick statistical analysis techniques available, and perhaps these  alone should have been up to the task.  But as the current liquidity crisis demonstrates, the devil is not only in the details of risk analysis.  It is in the failure to take these probabilities to heart.

The concept of Enterprise Risk Management, or the incorporation of risk assessment in all functional areas of an organization, is not especially new. In 2002, for example, Sarbanes-Oxley required internal controls on financial reports, usually including risk assessment. However, this tidbit from an article on “Smarter Risk Management” from Director of Finance Online caught our attention:

“Standard&Poor’s has recently indicated that they will begin incorporating consideration of the strength of enterprise risk management practices as a component of their credit ratings methodology. This is yet another incentive for ensuring that a company’s approach to risk management is robust, capable of being articulated and will stand up to scrutiny.”

If your S&P score depends, at least in part, on your risk management methods, there may be hope for us yet. There are others signs that enterprise risk management is making a comeback, or at least is remaining in the corporate consciousness. A September piece from Business World Online indicates:

“Risk management has recently come into prominence in the corporate suite. … Risk management exists because a company wants to take advantage of or minimize risks that affect it. These factors include political risks, foreign exchange risks, interest rate risks, liquidity risks, price risks, market risks, operational risks, credit risks, and employee risks.”

What many of these reports miss, however, is the value in learning about ERM directly from others facing risk – and not just in your own industry. The Palisade Risk&Decision Analysis Conference in New York City (Nov 13-14, 2008), is an example of a risk forum bringing together executives from a variety of industries for the purpose of exchanging ideas on risk. Over 20 case studies form a key learning model, along with software training.

DMUU Training Team

The Director of the European Offices of the International Monetary Fund, Saleh M. Nsouli, recently gave an address that examined some of the lessons that can be learned from the crisis in today's world financial market. In both the private and public sectors, Nsouli advises taking a harder look at the risk management sector.

For the private sector, Nsouli advises that risk analysis not be ignored, even when profits are up:

"The governance structure of the risk management system needs to be improved in financial firms in which the incentives are biased toward returns rather than the risks involved in attaining them.

"Compensation schemes in many organizations focus on returns and, for the most part, ignore the risk taken to obtain such returns. The risk managers, because they are not profit centers and do not sell products or write trading tickets, tend to be ignored when profits are up. Indeed, many of them apparently did sound the alarm bells before the crisis set in and were often disregarded as too out-of-touch with new structural trends, though not all firms downplayed the advice of their risk managers. The key is to ensure that top management hears both sides at equal volume, choosing the risk-return combination which best represents the risk appetite of the firm."

And in the public sector, Nsouli identifies shortcomings in risk management systems as a key lesson:

"Supervisors and regulators need to have the incentives and resources to look hard and deep at possible flaws in the risk management systems of the institutions they oversee.

"Often, stress tests did not stress the right areas or not enough; funding liquidity risks received inadequate attention; and holistic views across credit, market, and funding risks were not emphasized in part because of the recent and constant attention on Basel II regulations, covering primarily credit risk."

These lessons point to the centrality of risk management, whatever the state of the current market. At many failing banks, there are risk management specialists who have had a frustrating experience in the last few months. Hopefully lessons of this financial crisis will be taken to heart, and credence will be given to proper risk analysis even when profits are up.



To learn about the latest techniques in risk and decision analysis, and network with top-level consultants, industry practicioners and Palisade experts, consider the 2008 Risk & Decision Analysis Conference, November 13th and 14th in New York City.

» More about the Conference