Risk and Decision Analysis Today

Holly Bailey

Public Relations Representative for Palisade

I specialize in communications for technical and scientific companies. During my work for Palisade Corporation over the past decade I have kept a close eye on trends in quantitative decision-making techniques. I'm keeping this blog to report where and how I find these techniques--such as risk analysis, risk optimization, decision analysis, neural networks, and statistical analysis--being applied.

Rumors of Death

Monday, March 15, 2010 by Holly Bailey

Allan Roth, who writes a blog for CBS Money Watch called "The Irrational Investor," recently asked his readers a rhetorical question: Is Financial Monte Carlo Simulation Dead? Since rhetorical questions demand an answer in less time than it takes the questioner to draw breath, Roth obliged.

While expressing sympathy for the investors who were victims of poor risk assessment and forecasting when the financial markets shook themselves down to rubble in 2008, Roth is taking a very politely defensive swing at one of the many critics of risk analysis who have turned up the volume since then--one Jim Otar of Otar Retirement Solutions and the author of Unveiling the Retirement Myth.

Roth is an experienced user of Monte Carlo software who knows the pitfalls of overoptimistic assumptions. He says he finds 99 percent of the Monte Carlo models he's see over the years to be inadequate because of this flaw. Jim Otar, for his part, finds other flaws as well: in the generation of randomness and trends and in the sequence of returns. Otar's modeling method does not rely on randomness but on a century's worth of historical data.

Our two worthy opponents put their models up against one another in a match that crunched identical inputs. Their models produced very, very similar results, apparently satisfying each analyst as to the superiority of his method. But while Roth said nice things about Otar and his model, he pointed out the limitations of relying on historical information alone. In other words, he doesn't concede.

For any kind of retirement planning models, he says, the cure to flaws is conservative input. Then he giddily sends his readers to one of those rudimentary online Monte Carlo calculators that investment firms love to offer their clients.

Rumors of this death are greatly exaggerated.

Confusion, Consensus, Certainty

Thursday, March 11, 2010 by Holly Bailey

Longtime user of Palisade's Monte Carlo software and other decision analysis tools, Willy Aspinall uses these tools to beat back some heavy-duty varieties of uncertainty. How long will it be before a volcano actually blows its top as opposed to gurgles over its rim? What factors should transportation officials focus on to reduce the likelihood of airline disasters? What are the acceptable limits of air pollution? What exactly will the climate be like for our grandchildren?

Aspinall is often called upon to provide expert testimony on these kinds of life-and-death questions, and he has recently called attention to one of the problems with expert testimony, including his own: In which expert should you place your confidence? In an opinion piece in this January's Nature--a magazine that is an icon of scientific validity--Aspinall describes the benefits of using a method called "expert elicitation" to balance the opinions of a group of experts. The method, developed by Roger Cooke of Resources for the Future, attempts to quantify and then pool the uncertainties to arrive at what Cooke calls a "rational consensus."

When experts disagree, Cooke has pointed out, any attempt to impose agreement will "promote confusion between consensus and certainty." In order to get around this problem, Aspinall points out in his article, the goal of risk analysis should be to "quantify uncertainty, not to remove it from the decision process." His ongoing risk assessment of volcanic activity on the island of Monserrat in the West Indies is the longest running application of Cooke's "expert elicitation" method. For details about how the elicitation and the pooling of opinion works, I recommend taking a look at the January 2010 issue of Nature.

Opportunity Costs of Risk Analysis

Friday, February 12, 2010 by Holly Bailey

Merck's Art Misyan, currently Director, Financial Evaluation and Analysis at the company and a longtime practitioner of risk analysis and decision evaluation, has offered some cogent comment in response to my blog about the calculating the opportunity costs of risk analysis in making decisions under uncertainty:

"In the Vail Daily News comment, they refer to the cost of being the second entrant. The impact of losing your innovative advantage can be somewhat quantified in a sales forecast, for example: if our launch is delayed, or if we are no longer the first entrant, then there is an EPS impact of $X.

For day-to-day risk management activities, quantifying opportunity costs is more challenging. Sometimes the best decision is the one you didn't make, and other times it costs you either in ongoing transaction costs, deal premiums, etc. For example, transaction costs can rise if the markets become more illiquid over the course of a trading day (say you're trying to trade Far East currency, but now it's late in the day Eastern Standard Time). Or, if you are executing a large-sized deal but don't place the order until late in the day - and the trade has to happen. So, hypothetically, you could calculate the impact of transaction costs, based upon average deal size and bid/offer spread at a time of day.

As a finance representative on the deal team, you are trying to help management arrive at quick decisions with the best available information, while understanding the potential risks. You don't want to be the "speed bump" in the process (again, very difficult to quantify). As part of the economic analyses, we summarize as many risks as possible, as well as a list of potential events that could impact our assessment. After management has reached a decision, we will revisit the numbers if or when these events occur over the course of the due diligence process."

Words from the wise to the wise.

The Rise of the NOMFET

Friday, February 5, 2010 by Holly Bailey

By now we've become accustomed to the marvels of neural network technology and, in fact, inured to the advances it brought in statistical analysis with its computational simulations of nerve cells. Its many everyday applications--especially in online retailing--seem kind of ho-hum, and we'd be put out if for some reason they weren't in use. Wasn't it only four or five short years ago that neural nets themselves were big news?

Last week there was more big news about neural networks: a French research team's announcement of an "organic" transistor that mimics a brain's synapse. Neural network computing is based on computational stand-ins for biological neurons, and linking these neurons with electronic synapses currently requires at least seven transistors. One new "organic" transistor can take the place of those.

The key here is nano. Tiny. Tinier than tiny. The new transistors are made of nanoparticles of gold and pentacene on a plastic substrate. The resulting connector is called a nanoparticle organic memory field-effect transistor: a NOMFET.

Not only will the NOMFETs accelerate the performance of neural network circuits, but because the human brain uses 10 to the fourth times as many synapses as neurons, the space saving NOMFETs will help make possible a generation of computers inspired by the human brain.

The rise of the NOMFET may also make possible another kind of advance, one that I find a little scary to contemplate. Because its built on plastic, the NOMFET could potentially be used to link a computer with living tissue. Get back, Frankenstein.

Goldilocks Had It Easy

Monday, February 1, 2010 by Holly Bailey

Ed Biernat, Consulting with Impact, has been in touch to respond to my recent question about analysis paralysis: How do you know when you've done enough decision analysis, no more, no less than will benefit you?

Here's Ed's take on the issue: "Goldilocks had it easy. She eventually got it right the third time. This issue is one that we wrestle with in Lean Six Sigma overall, because it is easy to become enamored with the analysis of data. Analysis paralysis kills the speed of an implementation and must be vanquished at all costs. Inertia is the biggest foe that we face in implementing Lean Six Sigma. It was one of the big problems with the old model with statisticians in businesses (and why it is hard to find a pure statistician around now in anything but actuarial endeavors.) What the issue really comes down to the basic question, What Problem Are You Solving?

Golf makes a quick analogy. Let’s take the greatest 7-iron player in the world. This person can play the 7-iron like nobody’s business. In fact, they use the club more than any other club in their bag, and crowd really appreciates this virtuoso of the 7-iron. But what is the purpose of the game? To use the 7-iron or to get the lowest score on the course? For risk-analysis geniuses, we can substitute the risk analysis tool for the 7-iron. It is a great tool, a powerful tool. But only if it helps us solve the problem we are facing. And that problem is probably not to build the world’s best model.

If you have addressed the question that you started with when you built the model, then you have done enough analysis. In our consultancy, our bias is to get close and move forward unless we are dealing with a mission-critical decision. We fully admit that we are not modeling experts, and we are OK with that. That is not why our clients engage our services. We solve problems and help them to change their culture. Modeling helps with that by getting the team familiar with issues and sensitivities before we do a full deployment. Once they can see the impact of this variation and their assumptions, and once they have a framework for going forward, we put the model away because it's done its job."

Thanks, Ed, for giving this some thought!

Cost-Benefit Feedback Loop

Friday, January 15, 2010 by Holly Bailey

An anonymous comment in the Vail (Colorado) Daily News about the dangers of overanalyzing a decision reminded me that, while the benefits of risk analysis have been much vaunted, the costs of decision evaluation have not been clearly defined. Sure, it's pretty easy to come up with a figure for a DFSS training effort or a budget for an entire risk management department. But what about the statistical analysis process itself?

Well, there's staff time or your own time (which is worth something), Monte Carlo software, some portion of your computing costs,data acquisition, and on and on. Many variables. But the kind of costs I'm thinking of are the kind you rack up while you're analyzing, say, option valuation, and not doing something else. These are opportunity costs. They are what really limit how thoroughgoing your risk analysis becomes, which layer you drill down to--and they are very difficult to quantify.

How do you calculate whether the time you're spending in risk assessment is cost-effective? It's a problem of operations risk. So I suppose you could enumerate all the other activities that would consume the same amount of time and model their paybacks. But that would cost you more time in statistical analysis. . . . and you would be left in a positive feedback loop.

In the days ahead I'll be talking to risk management and operations research folks to find out how they decide how much analysis is just the right amount--not too much, and not too little. I'll be surprised if I turn up any computational approaches--but who knows?

Predicting Customer Will

Tuesday, January 12, 2010 by Holly Bailey

If hindsight is twenty-twenty, foresight--at least in the world of market research--still has a ways to go. Simulation, both with Monte Carlo software and with a conjoint simulation approach, has been used by market researchers for some time now. Recently David G. Bakken,who maintains a blog on the Smart Data Collective site, pointed out that the drawback of these models is that even those that incorporate random number generation are static. That is, the inputs and the coefficients determine the model outcomes.

What's wrong with deterministic models? Nothing, I gather, except for the limitation that those that are applied to marketing research questions tend to treat the target customers, the companies devising product strategies, and their affiliates in advertising and PR as blocs that make decisions without benefit of individual will.

Agent-based models, which were born in the social sciences, simulate the interactions of multiple players, each of whom will act, absolutely rationally, in his or her own best interests. Bakken believes that agent-based modeling used in tandem with traditional risk analysis models or evolutionary programming methods such as genetic algorithms, offers a more dynamic means of accounting for the future behavior of potential customers.

On the face of it, Bakken's proposal seems to have merit. If the technique works for the social sciences, maybe it will work for marketing research. After all, what is marketing if not a commercial application of social science?

A Downturn for the Better

Thursday, January 7, 2010 by Holly Bailey

Honoring a time-honored tradition for the turn of the year, I've been looking back over the year just past to do a little retrospective trend-spotting. Here's one that took me by surprise: in spite of the downturn in the economy, there was also a downturn in online fraud. It's counterintuitive--historically, hard times are correlated with rising crime--but apparently true.

Late last year, DigitalTransactions, an online publication catering to businesses engaged in the "electronic exchange of value," reported that the results of a survey of principals in these businesses showed an overall decline in fraud of about 1 percent.

The survey, sponsored and carried out annually by a California risk management company, is the first in its eleven-year history to show a fraud rate this low. In 2009 North American merchants were expected to lose (a mere) $3.3 billion, in contrast to their loss of $4.0 billion in 2008.

What's behind this good-news downturn? Probably not increased honesty. There was no data on attempted fraud, and the assumption is that the increased use of automated fraud detection tools cut the merchant's losses. The level of sophistication of these tools has ratcheted up to the level where neural network classification, risk analysis, and statistical analysis of correlated data can take place in real time during the processing of a transaction. Furthermore, the combination of operational risk software with device identification of the purchaser's computer now make it difficult for a single computer to mob an online merchant with multiple bogus orders.

So the good news is not about improvements in human nature. It's about improving the defenses of this booming sector of the economy.

2010: A Model Year for Risk Analysis

Thursday, December 31, 2009 by Holly Bailey

Resolved for 2010:

• No more risk assessment on the backs of envelopes.

• Take time for statistical analysis of past experience.

• Use decent Monte Carlo software.

• Choose variables wisely.

• Consider carefully the implications of probability distributions.

• Continue decision evaluation even after the chosen course of action begins.

• Revisit, rerun, and adjust model frequently.

• Make better decisions by the numbers.

• MAKE MORE MONEY.

The CDO Is Back in the Spotlight

Thursday, December 24, 2009 by Holly Bailey

About this time last year the term "CDO" began to make regular appearances in the news.
The so-called "Collateralized Debt Obligations" were commonly blamed for sending an already shaky finance sector into exponential decline.

Today "CDO" returned to the front page of the New York Times in article reporting an investigation by Congress, the Securities and Exchange Commission, and the Financial Industry Regulatory Authority into the question of whether Goldman Sachs and other investment banks that sold the CDOs engaged in dirty dealing against the clients who bought the synthetic debt packages. The concern of the investigators is that Goldman, Deutsche Bank, Morgan Stanley and others knew that the CDO investments would sour and profited from short selling the stock of companies that bought the investments.

The investigation is still in its early stages, and those involved in it are playing zipper lips. Whether or not the investment banks broke any securities laws is still to be discovered. But in the meantime, I find the complexities of this kind of trading daunting and am fascinated to think about the minds that created the deals. How did the financiers decide what to charge for the CDOs, how to determine their value-at-risk, and, if they did sell short against their customers, when to make the trades? Obviously, in addition to some very finely tuned risk analysis and a great big Monte Carlo software package, a love of brinksmanship was necessary.

This is the stuff of paper chase novels. One former Goldman Sachs dealer has capitalized its on its sales potential with How I Caused the Credit Crunch--how much risk assessment was involved in that move!--and as it unfolds, the current Times story promises just as much page-turning fun.

How Cool Is That?

Sunday, December 20, 2009 by Holly Bailey

A sentence on global climate in a new bestseller has set off a storm of press activity: "Then there's this little-discussed fact about global warming: While the drumbeat of doom has grown louder of the past several years, the average global temperature during that time has in fact decreased." This is from Superfreakonomics, by Steven D. Levitt and Stephen J. Dubner, and, just a week after the book's publication, the statement drew published comment by number of climate scientists, who referred to it as "ridiculous," "a concerted strategy to obfuscate and generate confusion."

More to the point, the Levitt/Dubner statement caused Associated Press science writer Seth Borenstein to look into the numbers on climate change. Because "talk of a cooling trend has been spreading on the Internet," Borenstein sought independent statistical analysis from a number of expert statisticians and supplied them with NOAA's on more than a century's worth of data on ground temperatures and also with data from 30 years of satellite-measured temperatures. The satellite-based data are those often relied upon by the so-called climate change "skeptics."

When the statistical analyses were complete and showed an upward trend in global temperatures, one of the statisticians referred to the Levitt-Dubner statement as a case of "people coming at the data with preconceived notions."

Leavitt said his statement about lower global temperatures was based on "eyeballing" temperature data from recent years. Now I ask you--my friends whose business is statistical analysis, environmental risk analysis (or any kind of risk assessment)--how cool is that?

Swine Flu in the Rearview Mirror

Thursday, December 17, 2009 by Holly Bailey

Epidemiology has long provided jobs for statistical analysis jocks, and right now the big question in epidemiology is swine flu. How goes the war?

The Centers for Disease Control began tracking the progress of the disease in April 2009, with the first laboratory-confirmed case of H1N1. At the beginning of November a public health blogger responded to claims that predictions of a pandemic of Swine Flu had been exaggerated by pointing out that the CDC and the states stopped counting cases early in the pandemic and that even in the first wave of H1N1 there was significant underreporting.

Citing an article that appeared in Emerging Infectious Diseases, the blog explains a CDC/Harvard Medical School estimate of actual cases during the first four months of the pandemic (the explanation includes some very interesting detail on how epidemiologists try to get a grip on a very elusive population) . The researchers used Monte Carlo software in a rearview mirror approach that combined risk analysis with a multiplier effect, a technique from statistical analysis, that adjusted the analysis at each step in the case identification process.

Using this technique the researchers estimated that for every laboratory-confirmed case of H1N1, there were actually 79 cases. In other words, predictions of a pandemic were not hysteria. And while public health officials are unlikely to get hard data that will allow them to measure the actual extent and severity of H1N1 infections, no doubt without efforts to prepare for the virus, their rearview mirror methods would eventually tell a much grimmer story.

Digital Eyes on Alien Life

Wednesday, December 9, 2009 by Holly Bailey

University of Chicago geoscientist Patrick McGuire has big plans for Mars. Previously he worked on an imager for a Mars orbiter that could identify different types of soil and rock by detecting infrared and other wavelengths, and now he is drawing on that experience to develop a space suit with digital "eyes" and a neural network that rides on the hips of the spacesuit and can sort out living biological material from other matter.

The digital eyes will detect and plot colors, and the neural net, which is known as a Hopfield neural network, will compare these color patterns to a database of information previously gathered from that area of planet in order to make an animal-vegetable-mineral determination.

This complex AI system has already been tested at the Mars Desert Research Station in Utah, and McGuire and his colleagues were satisfied that the Hopfield algorithm could learn colors from just a few images and could recognize units that had been observed earlier.

McGuire's concept is that a human wearing this neural network could simply walk around the red planet and record every nearby object, rapidly gathering information.

Obviously, such a clothing item awaits a manned Mars mission. But in the meantime, why not have the next Rover suit up?

Monte Carlo, Where Speed Counts

Saturday, December 5, 2009 by Holly Bailey

Apparently the real test of computer chip performance, that is, speed, is spreadsheet simulation. PC Magazine blogger Michael Miller recently published a comparison of four new computer chips, two form Intel and two from Advanced Micro Devices. Interestingly, Miller was not comparing the two similar notebook computers running these chips, just the chips themselves.

Miller put the chips through a number of tests and noted certain ups and downs in performance. By the clock the chips ran at the same speed, but speed varied according to the kind of application (Miller doesn't actually name the spreadsheet software, but it seems a safe guess that he's using Excel). For Miller, what really sorted the good from the best, the merely speedy from the truly fast was running Monte Carlo software, especially running big models based in huge data sets--the kind of simulations that typically come up in energy distribution and reserve estimation and operations management in oil exploration and production.

So which chips win the Monte Carlo Excel Grand Prix?

I'll defer to Mr. Miller, whose blog is loaded with interesting details.

The Cat is Out of the Bag

Thursday, December 3, 2009 by Holly Bailey

At November's supercomputing conference in Portland, Oregon, IBM announced that its researchers working with a team from Stanford University had succeeded in developing an accurate simulation of human brain function. The simulation will be capable of emulating sensation, perception, action, interaction and cognition.

This algorithm simulating a living neural network, called BlueMatter (spelled as one word like everything else in computerese these days) is an important milestone in IBM's mission to build a cognitive computing chip because it begins to advance large-scale simulation of a cortical neural network and it synthesizes neurological data. BlueMatter is built with Blue Gene (two words for this pun in the singular) architecture, which, in combination with specialized MRI images, allowed the team to create a wiring diagram of the human brain. This map of the brain is, according to IBM's press release "crucial to untangling its vast communication network and understanding how it represents and processes information."

To be more accurate, what BlueMatter has thus far demonstrated is the potential to achieve neural network technology that operates on the scale of complexity of the human brain. The algorithm's current simulation approximates the cortical system of a cat. Hence, the title of the paper announcing IBM's accomplishments: "The Cat Is Out of the Bag." Even so, this is an operations research accomplishment that dwarfs such mundane analytical tasks as option valuation, value-at-risk, or reserve estimation.

One of the goals of the company's cognitive computing program is to create a chip that operates with the energy efficiency of the human brain (20 watts). But in order to emulate the brain activity of a cat, the research team had to bring out one of the largest supercomputers in the world, the IBM Dawn Blue Gene/P--which comprises about 150 thousand processors and contains 144 terrabytes of main memory.

This cat came out of a pretty big bag.

Two Sides of the Coin

Wednesday, November 18, 2009 by Holly Bailey

Maybe it's because of fallout from the past year's financial crisis, but I have been noticing that almost all the press mention for risk analysis or Monte Carlo simulation is in connection with fending off the bad stuff--loss, adversity, or failure of various kinds. So it was refreshing to come across a story of decision evaluation being used to analyze the good stuff, that is, innovation and opportunity.

In 2008, Dell sponsored a student team from the Tauber Institute at the University of Michigan to compare the opportunity scenarios for designing new laptops that would use emerging wireless technology. Dell's challenge to the engineering and business students was to determine the most profitable way to approach new laptops for new markets.

Out came the laptops, out came the Monte Carlo software. In went the inputs--the possible cards, the cost of components, retail discounts vs. direct sales, necessary changes in internal organization. What was the value-at-risk? An already pretty profit picture from the laptop sales of the previous year.

It was the most positive kind of problem to solve. And what was the outcome of the team's efforts? "A Profit-Based Simulation Model for Laptop Planning"-- an optimistic title if there ever was one. But I suppose the title could have been "Modeling Potential Loss from New Laptop Design." There were quite a number of good-news scenarios at the institute that year. I mention the Dell team because of the intensive decision analysis element.

As anyone who does risk analysis is aware, the flip side of opportunity is risk, or maybe opportunity is the upside of risk. They are always there together, the two sides of chance, but it's great to occasionally see the brighter side of the coin.

Monte Carlo Meets Monte Carlo

Thursday, November 12, 2009 by Holly Bailey

Monte Carlo is known not only for its casinos and the games of chance that are the namesake of the risk analysis method but also, just as famously, for motor sport. Now, although this has been very little publicized, it appears that Monte Carlo meets Monte Carlo, on a regular basis.

A couple of weeks ago, a news item from the United Arab Emirates tipped me off to the fact that Formula 1 racing teams include--in addition to drivers and pit crews--a panel of race strategists. It is the strategists' job to try to plan advantageous responses to any eventuality in a race--rain, wrecks, repairs. Even with the help of computers, forecasting all possible scenarios for a single race is a full-time job, and the F1 strategy teams rely heavily on their Monte Carlo software.

Risk analysis began contributing to F1 strategy as far back as the 1990s and was credited for the McLaren team's 2005 victory in the Monaco grand prix. It is now standard operating procedure. Strategy teams not only pre-play every corner, every curve of a race circuit, but even after the start has sent the cars into high speed, the strategists are responding minute by minute to action on the circuit by running new risk assessments and statistical analyses of emerging scenarios and sending their advice for the drivers via high-speed data links.

Although the race strategist squads haven't received much press, their presence makes perfect sense. After all, who does more and faster decision making under uncertainty than a race driver? And what about the engineers who fine-tune features like aerodynamics and brake design in preparation for a particular race course? And the pit crews on race day? Their function is life or death operations management.

It's a deadly game of chance, the perfect venue for Monte Carlo to meet Monte Carlo.

Risk Studies

Wednesday, October 28, 2009 by Holly Bailey

This month geographers Pierpaolo Mudu and Elise Beck put out a call for papers for the next annual meeting of the Association of American Geographers. Their session will focus on the "social geography" of risk. Social--or human--geography is devoted to identifying cultural, political, and economic patterns that play out on the physical landscape.

Needless to say, every item in the list of session topics is a virtual Pandora's box of risk. Here are a few of them: "natural" versus "non-natural" risk, perceptiion of risk, environmental risk analysis, different scales to map risks, vulnerability modeling (which I assume was comparative risk analysis). There is lots of juicy fodder for folks who enjoy taking aim at uncertainty with their computers, especially because building models that address these topics often involves integrating GIS techniques with Monte Carlo software.

While I am intrigued by the almost unfathomable risks outlined in the call for papers and the thought of all those number-crunching social scientists who have only six months to plumb the depths of these topics, I was even more intrigued by the mention of what is apparently a new emerging academic field. It's called risk studies--something like American studies, only for quant types who want to get to know the lay of the land.

A Random Walk

Friday, October 23, 2009 by Holly Bailey

It has struck me often that the target of statistical analysis writ large is randomness. Because it is the fundamental concern of all the techniques for decision making under uncertainty that I try to track--risk analysis, genetic algorithm optimization, and neural network prediction--I assumed I understood the term randomness pretty well.

But it turns out that what you think randomness is depends upon what line of work you're in. If you are a statistician, randomness occurs in a repeating process the results of which follow no discernible pattern. If you are a geneticist, randomness applies to genetic mutations that are not controlled by genes. If you are a financier, randomness is blamed for controlling stock prices, which respond instantaneously changes changes in available information.

In all these fields, randomness is closely allied with chance and probability, and the human struggle against chance is epic. We resort to sharper and sharper tools to pare down what appears to be random, and just when we think we have the magic tool--say, Monte Carlo software--or the magic idea--the random walk hypothesis, for example--the definition of randomness changes.