Can “expert” training increase confidence while making judgments worse?

I came across more interesting research about possible “placebo effects” in decision making.  According to the two studies cited below, receiving formal training in lie detection (e.g. so that law enforcement officers are more likely to detect a untruthful statement by a suspect) has a curious effect.  The training greatly increases confidence of the experts in their own judgments even though it may decrease their performance at detecting lies.  Such placebo effects were a central topic of The Failure of Risk Management.  I’m including this in the second edition of How to Measure Anything as another example of how some methods (like formal training) may seem  to work and increase confidence of the users but, in reality, don’t work at all.

  • DePaulo, B. M., Charlton, K., Cooper, H., Lindsay, J.J., Muhlenbruck, L. “The accuracy-confidence correlation in the detection of deception” Personality and Social Psychology Review, 1(4) pp 346-357, 1997
  • Kassin, S.M., Fong, C.T. “I’m innocent!: Effect of training on judgments of truth and deception in the interrogation room” Law and Human Behavior, 23 pp 499-516, 1999

Thanks to my colleague Michael Gordon-Smith in Australia for sending me these citations.

Doug Hubbard

Still no evidence…

I get many emails about the validity of Risk Maps, Risk Scores and Risk Matrices in risk analysis.  I know there are a lot of passionate proponents of these methods (and many heavily-invested vendors) but there is still no evidence that such methods improve forecasts or decisions.

Read my skeptical look at many risk analysis methods in the OR/MS Today and Analytics articles (See the Articles page).  I compiled quite a lot of research in these articles and even more in my book that should lead any manager to be suspicious about whether the perceived effectiveness of these methods is real or imagined.

I’ve updated the upcoming second edition of How to Measure Anything with even more research in that vein.  Stay tuned and post your comments.

Doug Hubbard

Evolving Taxonomy of Major Risks

Originally posted at http://www.howtomeasureanything.com, on Friday, September 11, 2009 9:02:59 AM, by Unknown.

“On page 47 of HTFRM Mr. Hubbard refers to the website for viewing an evolving taxonomy of major risks.
I am having trouble locating it, am I missing something?”

Not at all. We haven’t had much discussion about it so it was evolving very slowly.

But this is where we wanted to posts those risks. I wanted to keep a running tab of major risks and have discussions about prioritizing them. Eventually, this would be part of a growing GPM.

The important thing is to have something that is not static and currently, there really is no “real time” list of risks. Part of the problem in risk management is that it seems to be about the last big disaster. After 9/11, RM was about terrorism, after Katrina it was about natural disasters, and after 2008 it was about financial crisis.

The news gives us some list of risks but those risks are not exactly chosen for their validity as real risks. Still, it gives us a starting point. While we don’t want to merely react to the latest unique catastrophe, we need to know that the catastrophe teaches us what is possible. And if we think about it, it can be extrapolated to other possibilities.

There are several things that concern me for the next few decades. But the theme I would like to start with are a category of natural disasters so large that a comprehensive response by emergency services is impossible. Below I list two to start with. While these are major national and global catastrophes, they should be considered from the point of view of individuals and organizations.

1) A major earthquake on the west coast of the US. The largest earthquake scenarios are many times larger than we would have the resources to respond to. Individual organizations should consider how to relocate operations in that area, arrange transportation of staff, and ensure continuity.

2) The one in a few-century tsunami happening slightly more often. Rising oceans from global warming (regardless of what anyone thinks the cause is, the rising oceans are a fact) will have two impacts that compound in a potentially disastrous manner. Slightly heavier loads on the ocean floor as the sea level rises will have seismic effects that can not be confidently modeled by geologists. The slightly deeper water has a non-linear effect on tsunami size for a given earthquake. A large wave from oceans just a few centimeters higher will be able to travel much further inland. Combine this with the fact that current population density on all the oceans shores is much higher than it was during the last once-in-200 years tsunami. As devastating as it was (100,000+ killed) the 2004 tsunami was not near a major metropolitan population like the west coast of the US. It is just as likely that the next big one will near the US as anywhere else.

Any more ideas out there? Let’s not just post the latest big concern on the news, but situations that are so bad that it would be hard to think through the consequences. If disasters like those two above happened – which is quite possible in the span of the current generation – organizations and individuals couldn’t possibly count on assistance from authorities. Either of these events could be many times worse than Katrina. And California, a major part of the US economy, is particularly vulnerable to these scenarios. Events like these seem like science fiction to most people who haven’t actually lived through something like that. As I mention in the book, the mere lack of such events recently doesn’t change the risk, but it does increase our complacency.

Thanks for helping to get this started,

Doug Hubbard