“If the initial assessment of risk is not based on meaningful measures, the risk mitigation methods—even if they could have worked—are bound to address the wrong problems. If risk assessment is a failure, then the best case is that the risk management effort is simply a waste of time and money because decisions are ultimately unimproved. In the worst case, the erroneous conclusions lead the organization down a more dangerous path that it would probably not have otherwise taken.” – Hubbard, Douglas W. (2009).The Failure of Risk Management: Why It’s Broken and How to Fix It.
5 comments
Trackbacks/Pingbacks
- Twitted by ellenjharris - [...] This post was Twitted by ellenjharris [...]
- Twitted by Simplecon - [...] This post was Twitted by Simplecon [...]
You must log in to post a comment.
Which book, by the way? Two of my three books address quantitative vs. qualitative. But my response is the same either way (I just need to know where to send you in the book).
The terms you use seem to imply that you’ve already made a judgment about the superiority of qualitative vs. quantitative methods. If so, you can produce the evidence for this position and if not the terms are presumptively dismissive. I can easily turn that same language around on your position and you will see what I mean. To borrow your terms, I might say “Instead of complaining about quantitative methods it is better to embrace it and understand it. Do not get hung up about the ambiguities of qualitative methods – focus on the probability.” If I did not present evidence for the superiority of quantitative vs. qualitative methods, this statement would also be presumptively dismissive. But I do present that evidence. If you get to the point where I cite that evidence and can present more convincing citations to the contrary, I would be interested in seeing them.
To address another part of the comment, I think some of your concerns will be eliminated later when you understand how measurement is actually defeined in the empirical sciences. (Which I address early in the book, How to Measure Anything). Once you understand this then you see how even putting “quantitative limits” on something is a measurement if it is based on some observation that reduced your previous state of uncertainty. But if you are even using phrases like “measurable (in a quantitative sense)” I would simply suggest you spend more time on that part of the book before you move on.
I have never seen a quantitative analyst who didn’t understand qualitative methods but I knew many qualitative analysts who base their entire opinion of quantitative methods on misunderstandings of basic quantitative concepts. For example, your claim that “…once events become rare quantifiation beyond plausibility becomes meaningless” would be be overturned once you familiarize yourself with methods for quantifying rare events. The idea that small numbers are meaningless would not only be worthy of earning the Field Medal in Mathematics (if such a proof were possible), it leads to nonsensical outcomes in the real world. There is a small probability that I will die as a victim of crime. But if actuaries didn’t know how to estimate small probabilities like that for large portfolios of policies, their reserve estimates would be far off the mark. Some of the biggest risks are unlikely but catestrophic events. I shudder to think risk analysts for the US power grid, new drug development, or airline safety would be so dismissive of measurements of important events. We measure things because we have to make informed decisions. If you assert projects A and B are not measurable then you have a less informed method of choosing between them when faced with limited resources. You can attempt to manage portfolios of decisions based entirely on gut feel but evidence shows that even simple quantititve methods are measurably better (see my citations on Meehl, Dawes, etc.)
In summary, the position I take on these issues is really not a matter of opinion. If the evidence I presented (especially in the last subsections of chapter 12 in How to Measure Anything and multiple chapters in the second section of The Failure of Risk Management) would have come out in favor of purely qualitative human intuition, I would have written a book about that and someone else would be posting a comment that I should quit “complaining” about quantitative methods and that I was “hung up” on intuition and qualitative scales. So, whichever book you are reading, read the whole book and come to your conclusions about what can and can’t be done with quantitative methods and the relative value of qualitative methods after you see the arguement.
Thanks for your input,
Doug Hubbard
I didn’t mean to generate too much heat; as I point out, I’ve only just begun the book (the Failure of Risk Management) and am enjoying it enormously. I spend a lot of time teaching Risk Management in finance (perhaps badly?) and I have always been concerned around the focus on Value at Risk and the apparent search for a mathematical Holy Grail which allows meaningful statements about tail events. I’m emphatically not someone who believes that qualitative is superior to quantitative, I simply don’t think that qualitative judgements are ‘inferior’ because they are subjective. Mathematical approaches (commonplace in Finance) have added enormously to our understanding of what is going on; but there is a danger of being blindsided by them. I recognize that a lot of work (EVT etc.) has been done around rare events and I hope I understand that attacking the ‘normal’ distribution is irrelevant because nobody really uses it anyway. My concern (which may be changed/amended/dissipated after more reading) is that too much effort is taken up by attempting to fine-tune the underlying method (get some distribution, assess a probability, find a ‘worst case’).This may be undermined by the fact that no distribution can ever be perfect (we cannot know the ‘generator’), and the dynamics of risks are such that management may require more narrative judgements (“what happens next?”) rather then acting according to a number of potential future snapshots. But I need to read more; I am well aware that this is simply an initial view. The book is making me think, which is no bad thing.
I’m simply responding to your comments on a factual basis. No heat is intended.
At no point do I say that qualitative judgments are inferior “because they are subjective”. On the contrary, I support certain subjective methods but I do so only because there is evidence to support their value. I think that you will find that my assertions are backed by evidence. I assert that risk management has failed and even very early in the book I explain what I mean by that. The most popular methods are used without any knowledge of (or concern about) the measurable performance of the method itself. In most cases it has not been measured and where it has the most popular methods appear to add known errors. Furthermore, the methods that have shown a measurable benefit are not widely used. This is not the same as claiming, however, that there is positive proof that every qualitative method is inferior to quantitative methods or even unaided intuition. And I make no such claim. I simply point out that using an instrument with an unknown performance is a key risk itself especially when some similar instruments or parts of the same instrument do have a measured performance and the measurement shows it to be inferior.
You seem to be presuming I make certain arguments I do not make and I do think your questions will be answered. You will find that I also criticize certain quantitative methods. But all of my positions are based on measured comparisons of alternative methods. As I quote George Box later in the book “All models are wrong, but some are useful”. Whether a model is the “truth” may be unknowable but we can know whether using one model measurably outperforms another model. The only attempt to “fine tune” a method is when it can be shown that a particular tuning has a measurable benefit. (i.e. whether a particular amount of fine tuning is “too much” needs to be based on an estimation) That is the only “meaningful statement” we can make about decision analysis of any kind including the part we call risk analysis. The counter position would have to be that we shouldn’t be the least bit concerned with whether decisions or forecasts are actually any better using one method or another. Given the size and importance of many of the decisions driven by risk management, that position seems unsupportable to say the least.
Doug
Thanks for the reply; I apologise because I am definitely jumping the gun by commenting when I have only just begun the book. I recognize that I’m in danger of sounding like a politician who complains about a film they haven’t watched. The “all models are wrong, but some are useful” quote is a favourite of mine because it seems to emphasise the importance of judgement. I was also impressed with your initial question along the lines of “How do you know it works?” as this is indeed the (multi-)million-dollar question. Difficult to answer as the fact that I’ve never been attacked by a tiger doesn’t mean that my “anti-tiger attack” blue shirt is necessarily a reasonable mode of defence, irrespective of any number of historical measurements.
I need to read more – I’m not attempting to set up a straw man although I may be doing so unintentionally. I just have an innate suspicion of the “if it can’t be measured it can’t be managed/judged” style of thinking. I am not claiming that your book follows this approach because I don’t have enough experience of the book yet! I was just unnerved at the beginning where I recall a passage which seemed to suggest that the approval of an actuary would be better than using subjective management judgement. Note that this may well be a superficial mis-remembering, and I apologise.
Perhaps better that I read rather than comment.
Many thanks,
Peter Fraser
I do mention the “you can’t manage what you can’t measure” quote but I don’t assert that all management without measurement is ineffectual. I would, however, argue that the facts show that a wide variety of decisions are easily improved when augmented by measurements – including the decision about which methods are better. What I get in to later in the book is some of the research that shows how flawed our judgment can be when we rely on inution. These are based on the findings of multiple researchers each using a large number of subjects. When we see the kinds of errors which even seasoned experts include in subjective judgment, alternatives have to be considered.
I don’t recall exactly what I said about an actuaries opinion being better than subjective managment judgment in the beginning of the book, but I assure you I only state what the evidence allows me to state. In later chapters I describe evidence that shows how frequent certain kinds of errors are in subjective judgment. Experts routinely make incorrect inferences from data that would easily be corrected with simple calculations (see my analysis of the NASA study done by Robin Dillon and the work of Paul Meehl) and experts that rely more on quantitative methods measurably outperform experts who do not (the Fiona Lamb study of oil companies and my analysis of another set of NASA data). But we also know that subjective methods can be useful especially when experts are trained to provide subjective estimates of actual quantities (not “high, medium, low”) and that if certain errors and inconsistencies are controlled for (such as I mention in the Lens method) then expert estimates can be improved. We know that without these certain kinds of training and adjustments experts in all fields are systemically overconfident and highly inconsistent in their judgments. Even the error added by popular ordinal scoring methods has been measured (Budescu, Cox and others I mention). You will see all of those parts in the book.
V/R
Doug