Holly Bailey

Holly BaileyPublic Relations Representative for Palisade

I specialize in communications for technical and scientific companies. During my work for Palisade Corporation over the past decade I have kept a close eye on trends in quantitative decision-making techniques.  I'm keeping this blog to report where and how I find these techniques--such as risk analysis, risk optimization, decision analysis, neural networks, and statistical analysis--being applied.

The Better to Be Believed

Friday, August 27, 2010 by Holly Bailey
In his blog yesterday for Smart Data Collective, Dean Abbott, makes a worthy, commonsense observation: no matter how accurate a predictive model is, it is of no use to the enterprise unless it is presented in such a way that all the decision makers understand what factors and techniques went into the analysis and why.
 
The reason that the 'best understood' model is more effective than the 'best' model is that when the people with authority over a particular decision are presented with a statistical analysis that is beyond their ken, they may or may not pretend to understand it.  But in any event, they are not likely to buy into the results if they can't retell the story the model describes.  
 
Take for instance, a Monte Carlo simulation that focuses on credit risk analysis for a particular loan.   Everyone in the line of authority will be held responsible for real world outcome of what the Monte Carlo software describes in the Excel spreadsheet.   And if you are one of these decision makers, how can you take responsibility for something you may not quite understand?
 
The problem of acceptance of a predictive model presents the analyst with a tough question: Do I present the model that I know is true and statistically accurate?  Or do I present a ruder, cruder analysis that presents a story that can be immediately understood?
 
Abbott suggests a compromise: streamline your plot by masking (Abbott says "removing") fields that contribute to the robustness of the analysis but involve statistical twists and turns that are distracting to decision makers who may not be fascinated with technique and just want to see how the story turns out. This, he explains, allows you to work from a model both you and the decision makers can believe in.

Your thoughts? 

Rating the Polls

Monday, August 23, 2010 by Holly Bailey
With the New York State primaries coming up September 14 and the general election on November 2, I predict that as soon as summer turns the corner into September, we'll start hearing lots and lots about polls that predict election outcomes.  To find out if there was any early discussion of polls, polling, and outcomes, I returned to my favorite election forecast site from the 2008 presidential elections, FiveThirtyEight: Politics Done Right.
 
Sure enough, there it was, a comparative rating of pollsters. This will give people like me, who tend to believe any poll just because it's covered in the news, a way to assess the poll reliability. FiveThirtyEight is the brainchild of Nate Silver, and 538 is the number of members of the Electoral College.  Silver's primary business is Baseball Prospectus, which is also fueled by Monte Carlo simulation and other risk analysis techniques, but FiveThirtyEight has done well enough for the New York Times to want incorporate it in its online coverage during the coming elections.
 
Silver's grasp of statistical analysis becomes immediately evident when you go to his page on the pollsters, and he's more than happy to discuss the statistical methods he uses to rate the pollsters--regression analysis of raw data, Monte Carlo software in an Excel spreadsheet, weighting of poll performance data, and so forth. His take on these matters may be of practical interest to any of you who use these techniques in financial risk analysis.

Elections are all about decision making under uncertainty, especially voter decisions under uncertainty, and according to Nate Silver, only polls taken within 21 days of an election are reasonably reliable.  So when the national campaigns are ramping up in October, keep one eye on the polls and one on FiveThirtyEight.  



Taking the Price

Friday, August 13, 2010 by Holly Bailey
Everyone should be allowed at least one vice, and mine is horses.  I love them, spend as much time around them as feasible, and find that after years of this I'm still learning. Recently I've met a couple of people know a whole lot about horse racing.  They don't know a thing about the horse itself, but they have a very sophisticated understanding of the mathematics of predicting performance.
 
So that I could keep up my end of our conversations, I looked further into handicapping and discovered that horse races themselves are only a kind of graphical display to show the results of some massive efforts at statistical analysis, including some of the quantitative forecasting techniques used by financial analysts and whole lot of custom Excel programming.  This should surprise no one--after all, what is betting on a horse if not decision making under uncertainty?--but what did surprise me is level of technical discussion about the math and how to work it through in Microsoft Excel statistics.
 
Take a look, for instance at a recent blog on "taking the price" from the U.K.'s Simon "The God of Odds" Rowland.  Taking the price is locking in the odds when you bet.  He discusses how to correlate a horse's rating--the amount of weight the horse has been assigned to carry--with the actual odds on this competitor.  He then gives the mathematical recipe for his custom Excel spreadsheet, which combines Monte Carlo simulation and the related Markov Chains technique. He wraps up his demonstration with a standard disclaimer: "It must be immediately apparent that this process is very susceptible to the GIGO (garbage in, garbage out) principle. No manner of mathematical manipulation will make up for essential shortcomings in the ratings and in the confidence attributed to those ratings."
 
No matter how good your model, it's still You Play, You Pay.  And Rowland's disclaimer echoed a comment an influential racing veterinarian made to me: "Never invest in something that eats while you sleep."     

Prediction Markets

Tuesday, August 3, 2010 by Holly Bailey
Although they've been around for the last 20 years or so, prediction markets have begun to make news for their application in business operations. Heralded early on in books like James Surowiecki's The Wisdom of Crowds, prediction markets are a fascinating alternative to traditional forecasting methods, such Monte Carlo simulation, which extrapolate future events from past patterns.  Essentially a betting exchange where participants stake something on the accuracy of the information they offer up, a prediction market is a way of capturing emerging patterns. 
 
Prediction markets can be public or closed private exchanges, as in most business applications. Here's how it might work: a business sets up an online portal to gather intelligence from its employees on such issues as scheduling or production costs.  Each employee has a limited number of points to wager with the information he or she offers, and these points are value-at-risk, which means that an employee is likely to offer only information that is accurate enough to be worth the points. 
 
Why bother to play at all?  Darwinian competition.  With each winning piece of information, the participant gains collective respect.  Maybe he or she advances in rank on a leader board or maybe the company honors its top participants in a ceremony. 
 
While the accuracy of prediction markets is still a topic of some fairly warm debate in applied mathematics, a number of risk analysis services are concentrating their solution portfolios on predictive markets.  

A Little Limelight

Tuesday, July 27, 2010 by Holly Bailey
Limelight--and by this I mean positively glowing publicity-- shines only occasionally on quantitative analysis, and rarely on Monte Carlo simulation.  But there was, 6 years ago, Michael Lewis's Moneyball, which established a place for statistical analysis in major league baseball.  Now there is Relativity Media, LLC, currently one of the heaviest hitting movie production companies in the business, and, more specifically, there is Ryan Kavanaugh, its CEO, and Ramon Wilson, its executive vice-president of business development.
 
Two things about Kavanaugh and Wilson make them unusual: they are leading a movie production firm that is not only alive but growing, and they use quantitative analysis for lots of decision making under uncertainty.  What can be more uncertain than investing in a movie? Only somewhat unusual for the movie business is the fact that these two decision makers are under thirty-five--it's a youth oriented business--and maybe this is correlated with their emphasis on making decisions by the numbers.  
 
"You can't think of it as money," Kavanaugh has been quoted as saying.  "You have to think of it as math."  Given the multimillion-dollar budgets Relativity underwrites--the raw size of the risks involved--it's probably more comfortable for everybody at Relativity to think math.  The kind of math Kavanaugh is particularly devoted to is Monte Carlo simulation, and he talks quite openly about his company's use of it.  When it comes to variables, he names names: principal actor,  genre, director, release date, PG  or R, although in all probability (sorry), each of these variables is probably a set of variables.  
 
"Everything has to run on the principle of profit.  We'll never let creative decisions rule our business decisions.  If it doesn't fit the model, it doesn't get done."  That doesn't mean, he has explained, that if he really likes a project, he and Wilson can't juggle the variables to make the film project fit the model.  They change the parameters to reveal the path to profit.  And profit he has--the estimated assets of Relativity are about $2 billion.  

So Kavanaugh qualifies as a mogul, a math-for-movies mogul.  When the spotlight falls on him, Monte Carlo simulation isn't far out of it.

  

Introduction, by Way of Retraction

Friday, July 9, 2010 by Holly Bailey
Just after I posted my last blog questioning a recent Investopedia column in the San Francisco Chronicle, I had a congenial note from the author of that column, David Harper.  His column compared Monte Carlo Simulation with two other methods of calculating Value-at-Risk, and I was concerned that its view of risk and risk analysis techniques was overly simplified. David   was surprised to discover that column had just appeared because he wrote it five years ago!

The five-year lag explains a lot--Monte Carlo simulation was not nearly so widely adopted or carried about by so many software tools as it is today--and I should have suspected the article was a vintage piece before I started carping.

So I happily retract my concerns to introduce to you David Harper, CPA and certified Financial Risk Manager.  In response to my comment about the attitudes and techniques that led to last year's collapse of the financial markets, David says that, now that the black swan has flown, "the crisis should implicate both HistoricalSim VaR and parametric VaR (at least multivariate normal!) and point toward Monte Carlo Sim. I've been thinking for a while that all of this [I think he means lack of accuracy in specifying risk] should really boost Monte Carlo."

Investment commentary is only one of David's activities.  He is the founder of Bionic Turtle, a business devoted to e-learning about financial risk and preparation for the certification exam for financial risk managers. This is a worthy enterprise--I was relieved to discover that there are hoops financial risk managers have to got through to be called that--and for anyone who would like to know more about quantitative techniques for risk analysis, its website is worth prowling. 

Thank you, David, for setting me straight.  

Easy, But Is It Rigorous?

Friday, July 2, 2010 by Holly Bailey
Value-at-Risk is a calculation that predicts a worst case scenario in which the maximum loss for a specific investment would be realized.  Recently the San Francisco Chronicle investment blog Investopedia, ran a short series posts on VAR.  One of the more intriguing of these demonstrated three ways of calculating Value-at-Risk for a single stock investment for more than one time period.  
 
The three methods were historical simulation, variance-covariance, and Monte Carlo simulation. What was intriguing about the comparison of methods was the observation that best choice among these methods was the variance-covariance method because it was easy. The downside of using the historical method was the need to crunch data and the downside of getting out your Monte Carlo software--no mention of using historical data to inform your model--was that the Monte Carlo method was "complex."  
 
Does that mean that risk is simple enough to require only simple statistical analysis?  And doesn't this kind of thinking encourage financial planners to take a direct but drastically reduced view of the possible outcomes of an investment?  And isn't this the same turn of mind that led to the collapse of the financial markets only a year or so ago?
 
Variance-covariance assumes volatility only in terms of standard deviation, and volatility doesn't come in one flavor or standard deviation.  Neither does risk.   

Clear Legal Precedent for Dealing with Uncertainty

Monday, June 14, 2010 by Holly Bailey
A recent U.S. Court of Appeals case is timely not only because it involves corporate liability for ocean pollution when everybody in this country is morbidly tracking the BP spill in the Gulf but because it is a case in which the judge highlights and corrects some common misconceptions about Monte Carlo simulation.
 
In a consolidated case involving hazardous waste dumping in the Houston Ship Channel, the codefendants, Tenneco and Occidental, acknowledged liability for the  pollution cleanup, but they appealed a lower court's decision partly on the basis of the court's method of allocating costs. The court had called an environmental engineer as expert witness and statistical analyst.  The engineer used Monte Carlo software and court-established inputs for his model. The defendants challenged the court's inputs in the risk analysis model, and the Circuit Court decision rebutted their objections in clear terms.
 
Writing for the Fifth Circuit Court of Appeals, Judge Patrick Higginbotham said, "Monte Carlo measures the probability of various outcomes, within the bounds of input variables; to calculate Occidental's waste volume,. . .  Instead of simply averaging the input values, Monte Carlo analysis uses randomly-generated data points to increase accuracy, and then looks to the results that those data points generate. The methodology is particularly useful when reaching an exact numerical result is impossible or infeasible and the data provide a known range—a minimum and a maximum, for example—but leave the exact answer uncertain."
 
Responding to the charge that this method of statistical analysis is unreliable and untestable, Higginbotham responded,". . .the cited cases at most stand for the proposition that Monte Carlo analysis is unreliable when injected with faulty inputs, but nothing more. . . .  Monte Carlo simulation is not inherently untestable. . . . If anything, Monte Carlo provides greater certainty than the basic alternatives: using one of the three data or using the arithmetic average of all three."
 
Countering the challenge that the model results were "equivocal" the judge continued, " The Monte Carlo analysis—though it produced a statistical range of likely outcomes and not one determinative answer—supports choosing one result over another, and certainly assisted the district court in its decisionmaking."
 
The decisions-by-the-numbers guys certainly had their day in court.  The free advertising wasn't bad either.

Value-Based Management Compensation

Wednesday, June 9, 2010 by Holly Bailey
Full disclosure: I am, like so many of my friends, an investor––a small-time one--and recently, I have joined in the public outrage about bankers' bonuses and executive compensation in general. Compensation is one of the hot buttons in the debate over financial reform.  I keep wondering why compensation practices are what they are and how they could be adjusted to calm turmoil on Wall Street.

Enter Marwaan Karame, and his version of risk analysis.
 
Karame heads the New York consultancy Economic Value Advisors, which coaches major corporations on Value Based Management.  Value--long-term versus right-now profit--is the foundation of the firm's philosophy.  Its central principle is that any activity a business undertakes should increase the wealth of its shareholders--in the case of a privately held company, the number of shareholders may equal 1.
 
Karame has developed what he calls Value Based Compensation, and the goals of this are to align the self-interest of management with the self-interest of shareholders. He believes the shareholders, the company, come first.  And this means a lot of decision-making under uncertainty.  But Marwaan has a method for his management-shareholders balancing act, and it involves performance targets, statistical analysis, and risk assessment (in this case, managing probabilities of performance). His strategy involves maintaining a reserve of bonus funds and timing the payout of these rewards. 
 
The point at which Monte Carlo simulation and Monte Carlo software come in is the point at which variance between performance targets and the level and timing of reward converge. He shows his his client how to click into Monte Carlo in the Excel spreadsheet and use the software to locate the tipping point between wealth for management and wealth for shareholders. 
 
As a small--very small--shareholder, discovering that there is such a tipping point and that Karame knows how to locate it is reassuring.  Makes me feel there's someone on my side.   

Health Care Management: Decision Making at Two Levels

Tuesday, June 1, 2010 by Holly Bailey
Reading recent reviews of two books on healthcare caused me to realize that in spite of the rapidly increasing number of clinical studies that use risk analysis and neural networks to sort out the best treatment choices, there has been very little published on how to use quantitative tools like decision trees and Monte Carlo software to manage health care better. Given the recent national debates on health care reform, this is actually quite surprising. 
 
There's health care management, and then there's health care management.  On the macro level, decision evaluation focuses on the organization. Marian C. Jennings's Health Care Strategy for Uncertain Times (2000) prescribes ways for corporate health care managers to reshape the ways their organizations deal with uncertainty by adopting the same quantitative techniques used in the commercial realm by enterprises like investment firms and utility companies.  On the micro level, health care management focuses on you, your body. Thomas Goetz's The Decision Tree (2010) prescribes how to apply a number of these same decision analysis techniques to your own health. 
 
Essentially, what both books are saying is, "Look, the only certainty is uncertainty.  But you have some numbers.  Here are the tools to turn those numbers into plans you can reasonably rely on." These tools shouldn't be news to you as a reader of this blog, but apparently, if the popularity of Goetz's book and renewed attention to Jennings's are any indication at all, the health care management arena is plenty ripe for quantitative decision support tools.

Statistical Gizmos and the UK Election

Thursday, May 20, 2010 by Holly Bailey
The recent elections in the United Kingdom provided a really fun opportunity to see how extensively statistical decision evaluation and predictive modeling have penetrated popular culture.  The British press outdid themselves with online graphical gizmos that allowed readers to set the terms for outcome scenarios and let those spin out in true operations management style.
 
While The BBC offered an election seat calculator that really only translated voting percentages to number of Parliament seats won, the Guardian put up a Three-Way Swingometer.  With about 8, depending on which you count, parties in the fray, the Swingometer allowed readers to twiddle a dial to anticipate the effect of hypothetical party-shifting and election results.  
 
Next, Nate Silver, the election forecasting guru behind the FiveThirtyEight.com website, produced what he calls the Advanced Swingometer to offset the statistical disarray introduced by the original version's assumption of a uniform rate of "swing."  He backed this up with a demonstration of how elegant the statistical analysis  behind his model was. 
 
The Times came forward with a predictive map based on the predictions of gamblers in UK's lively betting shop scene.  Who know where those risk assessments came from.  
 
None of the online descriptions of the methods behind the gizmos were very detailed.  There were no mentions of named statistical analysis procedures, and this turns out to have been a good thing--because none of the gambits proved up to foreseeing the muddle that resulted from the actual voting.  If you wanted to try to come to a clear view of that, you will need to consult the decision tree posted by the BBC. 

Neural Nets Writ Small

Friday, May 7, 2010 by Holly Bailey
Of all the statistical analysis techniques I receive news alerts for, the neural network flashes up on my screen most often.  While I, like many of you, really enjoy the big-screen futuristic applications of neural nets--prediction of sun storms is a splendid recent example--there is a quieter trend ramping up at a more down-to-earth level. The nano level,that is the itsy-bitsy, teeny-weeny, the molecular level.  
 
For at least the past five years, the nanotechnology industry has been predicting and prototyping ways to incorporate neural networks into nano-machines.  This innovation has proved to be very handy for sensing devices.  The nano-sensor combines receptor particles with electronics controlled by a neural network algorithm.  The neural net sorts through the sensor responses to uncover patterns that trigger alerts.
 
This year there was a flurry of media attention focused on one of these sensing technologies, the nano-nose, which uses an array of nano-receptors coordinated by a neural network.  These sensors are being promoted to sniff out everything from explosives to disease.  
 
One indication of the expected adoption of applications that combine nano with neural is the advertising for neural network algorithms that can downsize to nano. But more than one of the nano-machine innovators has commented on the need to develop more robust statistical analysis techniques to improve the accuracy of the sensors.  Which means that there will be more neural network to shrink, which means that the algorithms advertised today may already be outdated.

Whatever the commercial considerations and no matter how blasé we become about technological possibility, there is still a big wow factor in packing a high-powered computing technique into such infinitesimal space, and you can be certain the nano people will be harnessing neural networks to many new kinds of more-mini-than-micro machines.

Calculating the There There

Friday, April 23, 2010 by Holly Bailey
Gertrude Stein once famously criticized Oakland, California because "there is no there there." I think of this often as I watch a fascinating and rapidly rising business trend, the conglomeration of social media enterprises via acquisitions and mergers.  This week the European company Wikio, a specialist in the "blogosphere and social media," acquired the French company Neotia, which purveys a "buzz monitoring and online reputation management platform." 
 
Wikio, which purportedly indexes a million websites, wanted to buy Neotia because that company is good at analyzing the influence of brands and buzz campaigns and because of its CEO's expertise in decision analysis. Neotia wanted to be bought because Wikio will provide potential clients a means of accessing Neotia. 
 
What is remarkable to me is that here is an industry in which the functions of the products are so new that their originators also have to originate names for what they do--buzz monitoring, measurement of online influence, and so forth--and yet with only a couple of years experience in most cases, these companies are buying and selling each other. How do they create product strategies for these products?  How do they calculate prices?  And then, when they want to swallow up a competitor, how do they calculate an offering share price and figure out their value-at-risk? And how do they project how long any of these values will hold?
 
Social media companies have to do a lot of decision making under uncertainty, a whole lot of uncertainty, because every risk assessment focuses on a brand-new kind of product with no history in its marketplace.  They have to bet fearlessly on a whole lot of unknown taking place on the immaterial Internet.  But as one who has enjoyed a brief two years experience working in the blogosphere, I'll bet that, as ephemeral as that realm may seem, there likely is some there there, some value when we figure out how to calculate it.  

20 Questions in a New Orbit

Thursday, April 15, 2010 by Holly Bailey
An Ottawa toy developer is trying to make a jet-propelled leap from an online game to space travel. His vehicle? A neural network designed as the back end system for a game of 20 questions. Twelve years ago Robin Burgener wrote a neural net program to train on the sequences of player responses to questions--beginning with Animal? Vegetable? Mineral?--posed by the neural network,              
 
 
The game is does more than pose simple yes-or-no answers to lead you to a conclusion. The neural network algorithm is able to pose different questions in different orders, and it gets the right answer about 80 percent of the time.                                                         , 
 
Now, apparently, the sky's the limit for Burgener's neural network.  He was scheduled to make a presentation late last month at the Goddard Space Flight Centre explaining the potential uses for a neural networked 20 questions on board a space craft. These uses center broadly on troubleshooting technical and equipment problems and subsequently anticipating future problems.  
 
If, as he claims is true, his neural net guessing program can work around responses that are misleading or downright lies, what that would mean for space travelers, he concludes, is that  "if a sensor fails, you're able to see past it."
 
I know what he means, I think, but I myself don't tend to look past sensors.        

Cost-Benefit Analysis in the Land of Buzz

Friday, April 9, 2010 by Holly Bailey
For the past couple of years, I've been following the advance of cloud computing into the marketplace.  Recently, as the Cloud has begun to--I can't say materialize as that might confer some notion of definable substance, which in this line of business is to be avoided at all costs--become a presence, information officers have been increasingly interested in matters of costs and benefits. Those who are considering migrating their current computing operations to the Cloud would like to make risk assessments that weigh CAPEX--capital expense--against OPEX--operating expense--and for that they will need help calculating the TCO--the Total Cost of Ownership. To forecast the TCO, they will need to get out the Monte Carlo software to predict their potential flow of data out through the Cloud, and depending on a company's familiarity with risk analysis, this "could = hire a consultant who understands the meaning of all this."
 
Recently, to help clarify matters, a Computerworld blogger declared, "The fact that people are so interested in cloud TCO indicates that the general value proposition of cloud computing has been accepted and absorbed."  The need for this incisive commentary he blames on the fact that "there's been an amazing amount of FUD"--Fear, Uncertainty, and Doubt--"strewn about on the topic of cloud TCO." 

My problem with this discussion, as you've probably gathered, is not the efforts of smart people to grapple with the opportunities and operations management issues raised by Internet-based computing.  It's the FUD that folks in computing seem to experience when it comes to clear, plain labels.  They flee into the land of buzz in order to assure TO--Total Ownership--of the terms.  
 
For starters, take the term Cloud for Internet.  It all gets just a little too. . . .well, vaporous.  It makes me feel like the grandmother of a man being ceremoniously installed as a dean at Cornell University a while back.  Having survived into her nineties and through the morning's pomp and circumstance, she asked her grandson what exactly he would be doing in this new job, and as he started to explain, she looked as if something tasted bad.  Finally she broke in.  "Honey," she said, "if you can't say it in one sentence, it has got to be illegal." 

Neural Nets vs. the Ripple Effect

Thursday, April 1, 2010 by Holly Bailey
About a week ago the Financial Times ran an article about a "new" investment analysis technique that could cut through turbulence in the financial markets: neural network analysis.  I thought okay, this isn't new but maybe the application is innovative.  Besides, I liked the metaphor the reporter used, a metal ball dropped in a vat of oil and the ensuing ripples that disturb the oil.
 
The article is about software developed by a Danish investment firm that turned its back on "linear" models to adopt a neural network approach that continually reclassifies investments in a portfolio and then makes suggestions about which equities to buy and which to sell. The proprietary software chews through a heap of data--prices, price-earnings ratio, and interest rates, for starters, and its performance bench mark is the Russell 1000 index. 
 
The test portfolio used to proof the method was acquired in 2007, just before the ball dropped into the oil.  For a time it seemed to hold up but then got caught in the turbulence and its undertow. It has now recovered nicely, ahead of the Russell 1000 in fact, and the asset managers are looking  for more investors. This is a sweet success story, especially given the demon turbulence looming over the project and the fact that the assets are apparently owned by the Danish state pension plan.

I understood the use of neural network software to counter nonlinear events like market turbulence, and I understood the continual classification and reclassification.  But I was intrigued that nowhere in the article was there a mention of risk, risk analysis, or even risk assessment.  Maybe it was there all the time, incorporated in the proprietary software, and maybe it just wasn't mentioned.  Certainly the asset managers who developed the program were aware they were at risk--they were chewing their nails as their fund slid down right beside all the other funds that were dropping in value.  But assessing risk doesn't seem to have been a factor in the firm's new defense against mayhem in the markets.  
 
So.  Is it time to shut down your Monte Carlo software?  I don't think so. . . .   

Rumors of Death

Monday, March 15, 2010 by Holly Bailey
Allan Roth, who writes a blog for CBS Money Watch called "The Irrational Investor," recently asked his readers a rhetorical question: Is Financial Monte Carlo Simulation Dead? Since rhetorical questions demand an answer in less time than it takes the questioner to draw breath, Roth obliged. 
 
While expressing sympathy for the investors who were victims of poor risk assessment and forecasting when the financial markets shook themselves down to rubble in 2008, Roth is taking a very politely defensive swing at one of the many critics of risk analysis who have turned up the volume since then--one Jim Otar of Otar Retirement Solutions and the author of Unveiling the Retirement Myth.  

Roth is an experienced user of Monte Carlo software who knows the pitfalls of overoptimistic assumptions.  He says he finds 99 percent of the Monte Carlo models he's see over the years to be inadequate because of this flaw.  Jim Otar, for his part, finds other flaws as well: in the generation of randomness and trends and in the sequence of returns. Otar's modeling method does not rely on randomness but on a century's worth of historical data. 
 
Our two worthy opponents put their models up against one another in a match that crunched identical inputs.  Their models produced very, very similar results, apparently satisfying each analyst as to the superiority of his method.  But while Roth said nice things about Otar and his model, he pointed out the limitations of relying on historical information alone. In other words, he doesn't concede.
 
For any kind of retirement planning models, he says, the cure to flaws is conservative input. Then he giddily sends his readers to one of those rudimentary online Monte Carlo calculators that investment firms love to offer their clients. 
 
Rumors of this death are greatly exaggerated.  

Confusion, Consensus, Certainty

Thursday, March 11, 2010 by Holly Bailey
Longtime user of Palisade's Monte Carlo software and other decision analysis tools, Willy Aspinall uses these tools to beat back some heavy-duty varieties of uncertainty.  How long will it be before a volcano actually blows its top as opposed to gurgles over its rim?  What factors should transportation officials focus on to reduce the likelihood of airline disasters?  What are the acceptable limits of air pollution?  What exactly will the climate be like for our grandchildren?
 
Aspinall is often called upon to provide expert testimony on these kinds of life-and-death questions, and he has recently called attention to one of the problems with expert testimony, including his own:  In which expert should you place your confidence? In an opinion piece in this January's Nature--a magazine that is an icon of scientific validity--Aspinall describes the benefits of using a method called "expert elicitation" to balance the opinions of a group of experts.  The method, developed by Roger Cooke of Resources for the Future, attempts to quantify and then pool the uncertainties to arrive at what Cooke calls a "rational consensus."
 
When experts disagree, Cooke has pointed out, any attempt to impose agreement will "promote confusion between consensus and certainty."  In order to get around this problem, Aspinall points out in his article, the goal of risk analysis should be to "quantify uncertainty, not to remove it from the decision process."  His ongoing  risk assessment of volcanic activity on the island of Monserrat in the West Indies is the longest running application of Cooke's "expert elicitation" method.  For details about how the elicitation and the pooling of opinion works, I recommend taking a look at the January 2010 issue of Nature.  

Opportunity Costs of Risk Analysis

Friday, February 12, 2010 by Holly Bailey
Merck's Art Misyan, currently Director, Financial Evaluation and Analysis at the company and a longtime practitioner of risk analysis and decision evaluation, has offered some cogent comment in response to my blog about the calculating the opportunity costs of risk analysis in making decisions under uncertainty:
 
"In the Vail Daily News comment, they refer to the cost of being the second entrant.  The impact of losing your innovative advantage can be somewhat quantified in a sales forecast, for example: if our launch is delayed, or if we are no longer the first entrant, then there is an EPS impact of $X.
 
For day-to-day risk management activities, quantifying opportunity costs is more challenging.  Sometimes the best decision is the one you didn't make, and other times it costs you either in ongoing transaction costs, deal premiums, etc.  For example, transaction costs can rise if the markets become more illiquid over the course of a trading day (say you're trying to trade Far East currency, but now it's late in the day Eastern Standard Time).  Or, if you are executing a large-sized deal but don't place the order until late in the day - and the trade has to happen.  So, hypothetically, you could calculate the impact of transaction costs, based upon average deal size and bid/offer spread at a time of day.
 
As a finance representative on the deal team, you are trying to help management arrive at quick decisions with the best available information, while understanding the potential risks. You don't want to be the "speed bump" in the process (again, very difficult to quantify).  As part of the economic analyses, we summarize as many risks as possible, as well as a list of potential events that could impact our assessment.  After management has reached a decision, we will revisit the numbers if or when these events occur over the course of the due diligence process."

Words from the wise to the wise.