A recent article in Bank Investment Consultant criticized the risk analysis method of Monte Carlo simulation for not taking into account extreme events like the stock market crash. According the article, a Morningstar executive states that the “bell-shaped curve that Monte Carlo simulations use” artificially assigns the probability of what happened as zero. Furthermore, the Retirement Income Industry Association calls for an update to Monte Carlo software simulators to include “a larger number of scenarios that assume greater volatility.”

These arguments demonstrate a fundamental lack of understanding of what Monte Carlo simulation is. The underlying assumption that Monte Carlo simulation itself is somehow to blame fails to recognize that Monte Carlo simulation is simply a mathematical technique that takes into account many different possible scenarios – but only within boundaries set by the user. You can’t change the underlying math behind these “what-if” calculations.

When modelers set up retirement planning or financial models, the people doing the modeling must make assumptions about the likelihood of different things happening – like the market crashing, for example. People may make those assumptions based on historical evidence or expert opinion, but it’s people who make those assumptions – not Monte Carlo simulation software. People then enter their assumptions into a Monte Carlo simulation model, setting up probability distributions to reflect their chosen likelihoods of occurrence. If the assumed volatility is insufficient, that is the fault of the modelers, not the simulation itself.

In addition, Monte Carlo simulation does not always “use” a bell-shaped curve. Uncertainty can be modeled with dozens of different probability distributions, many of them not bell-shaped. And the call to include more scenarios and volatility can easily be met by existing Monte Carlo software such as @RISK. It’s simply up the user to change the model parameters to look at more possible outcomes. The Monte Carlo simulation package won’t fight it.

It’s disappointing to see esteemed financial organizations such as Morningstar and the Retirement Income Association missing the point. Calling for changes to Monte Carlo simulation itself is not only impossible but fails to recognize the problems with modeling practices that led everyone to miss the crash.

Randy Heffernan

Vice President

I would like to clarify a point made by Randy Heffernan in his blog of June 4th on the Palisade web site. Referring to an article that appeared in Bank Investment Consultant, Mr. Heffernam writes:

[A] Morningstar executive [i.e.] states that the “bell-shaped curve that Monte Carlo simulations use” artificially assigns the probability of what happened as zero.

However, the article itself reads:

In the bell-shaped curve that Monte Carlo simulations use, “the probability of getting one of these extreme outcomes [like we saw last year] is basically zero,” explained Paul Kaplan, vice president of quantitative research at Morningstar, who notes that the Standard & Poor’s 500 Index has declined 13% or more in one month at least 10 times since 1926.

Mr. Heffernan placed the quotation marks in the wrong places, making it appear that I was referring to the Monte Carlo method itself rather than the assumption that asset returns follow a bell-shaped curve. The text that appears in quotation marks in the article itself is a direct quote from me that appeared in the Wall Street Journal on May 2nd. In that article I pointed out the problem with assuming a bell shaped distribution curve within a Monte Carlo model. I was not criticizing the Monte Carlo approach. Rather, I was making the same point made by Mr. Heffernan in his blog.

Sincerely,

Paul D. Kaplan, Ph.D., CFA

Vice President, Quantitative Research

You are absolutely right. Even esteemed organizations and many senior people often fail to understand that tools and techniques need to be used ‘smartly’ to be effective.