The Better to Be Believed

In his blog yesterday for Smart Data Collective, Dean Abbott, makes a worthy, commonsense observation: no matter how accurate a predictive model is, it is of no use to the enterprise unless it is presented in such a way that all the decision makers understand what factors and techniques went into the analysis and why.
 
The reason that the ‘best understood’ model is more effective than the ‘best’ model is that when the people with authority over a particular decision are presented with a statistical analysis that is beyond their ken, they may or may not pretend to understand it.  But in any event, they are not likely to buy into the results if they can’t retell the story the model describes.  
 
Take for instance, a Monte Carlo simulation that focuses on credit risk analysis for a particular loan.   Everyone in the line of authority will be held responsible for real world outcome of what the Monte Carlo software describes in the Excel spreadsheet.   And if you are one of these decision makers, how can you take responsibility for something you may not quite understand?
 
The problem of acceptance of a predictive model presents the analyst with a tough question: Do I present the model that I know is true and statistically accurate?  Or do I present a ruder, cruder analysis that presents a story that can be immediately understood?
 

Abbott suggests a compromise: streamline your plot by masking (Abbott says "removing") fields that contribute to the robustness of the analysis but involve statistical twists and turns that are distracting to decision makers who may not be fascinated with technique and just want to see how the story turns out. This, he explains, allows you to work from a model both you and the decision makers can believe in.

Your thoughts? 

One comment

  1. I absolutely agree with this – almost to the extent that the value of a probabilistic model is less about the actual number that comes out, and more about the discussions that occur as the team try to understand and buy in to the model’s assumptions.

    We put a lot of effort into communicating both the assumptions and the results of a model back to the team, and hence expose (and ideally resolve) disagreements within the team around the uncertainty that they have assessed. The model allows this discussion to happen, by providing tangible assumptions and logic that the team can challenge and discuss, and showing the consequences of these assumptions.

    However, a significant barrier (which we have managed to overcome) is that a probabilistic model’s results – usually percentiles such as P10, P50, P90 – traditionally cannot be broken down into its component parts. For example if a model consists of the cost of three components A, B and C, the P90 total cost does not break down into the sum of the P90 of each of the components.

    P90 of (A+B+C) ≠ (P90 of A) + (P90 of B) + (P90 of C)

    This statistical fact means that it is very difficult to answer management when they ask “how does our P90 spread amongst the individual components / months / work packages / investments …?” – without frustrating them with the response that “statistics don’t work like that”!

    The fact that we’ve been able to solve this problem means that we CAN now present probabilistic model results at a higher level of detail, and aggregated in different ways, so that the team and management can explore the consequences of the model, and ultimately understand how the model is behaving, even if not at the most detailed level. This makes the discussions much more fruitful and constructive, and gives confidence that the model is a reasonable representation of reality. It also allows much richer questions to be asked of the model, such as “what does the P90 cost look like over time?”.

    It is a big step further away from the common perception of a probabilistic model as a “black box”, whose results cannot be delved into, and can only be presented as the single overall result (e.g. P90) of the model. It has enormously improved our ability to communicate the model and get confidence, alignment and ownership from the whole team.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s