Comments for Hubbard Decision Research http://www.hubbardresearch.com The Applied Information Economics Company: Consulting, Training, and Execution Wed, 21 Dec 2011 13:49:28 +0000 hourly 1 http://wordpress.org/?v=3.3 Comment on The Measurement Challenge by stefano.palestini http://www.hubbardresearch.com/2010/03/the-measurement-challenge/comment-page-1/#comment-303 stefano.palestini Wed, 21 Dec 2011 13:49:28 +0000 http://blog.hubbardresearch.com/?p=491#comment-303 The likelihood comes from the observation of differences between the quartly financial position forseen and the actual, during 3 years (12 observation) I’ve seen 9 cases (70%) where the difference has been inside the range +/- 5 million and for 3 other cases has been +/- 20 million. Thank you in advance for your answer. stefano palestini The likelihood comes from the observation of differences between the quartly financial position forseen and the actual, during 3 years (12 observation) I’ve seen 9 cases (70%) where the difference has been inside the range +/- 5 million and for 3 other cases has been +/- 20 million.

Thank you in advance for your answer.

stefano palestini

]]> Comment on The Measurement Challenge by Douglas Hubbard http://www.hubbardresearch.com/2010/03/the-measurement-challenge/comment-page-1/#comment-302 Douglas Hubbard Wed, 14 Dec 2011 20:32:37 +0000 http://blog.hubbardresearch.com/?p=491#comment-302 Stefano, This sounds like a perfectly viable problem for the value of information. However, I would need clarification on something you said. It seems you are saying that the current accuracy of forecasts to actuals has a 30% chance of being within 20 million and a 70% chance of being within 5 million, right? That seems backward to me. The wider range should have the higher probability unless I misunderstand how you are stating this. The same would seem to hold for the target accuracy. But, that aside, improving the value of forecasts is certainly something the information value pertains to. You have a cost of overestimating and/or a cost of underestimating, right? How much would you lose for every million you over or under estimate? The product of this "loss function" and your probability distribution for the forecast is the expected opportunity loss (EOL). You have an EOL for the current state and for the desired target accuracy. The difference between the two EOLs is the value of information. Technically, you would only be computing EVPI if you were comparing the value of your current accuracy to perfect forecasts, which I'm sure is not what you mean. Thanks for your comment, Doug Hubbard Stefano,

This sounds like a perfectly viable problem for the value of information. However, I would need clarification on something you said. It seems you are saying that the current accuracy of forecasts to actuals has a 30% chance of being within 20 million and a 70% chance of being within 5 million, right? That seems backward to me. The wider range should have the higher probability unless I misunderstand how you are stating this. The same would seem to hold for the target accuracy.

But, that aside, improving the value of forecasts is certainly something the information value pertains to. You have a cost of overestimating and/or a cost of underestimating, right? How much would you lose for every million you over or under estimate? The product of this “loss function” and your probability distribution for the forecast is the expected opportunity loss (EOL). You have an EOL for the current state and for the desired target accuracy. The difference between the two EOLs is the value of information. Technically, you would only be computing EVPI if you were comparing the value of your current accuracy to perfect forecasts, which I’m sure is not what you mean.

Thanks for your comment,
Doug Hubbard

]]> Comment on The Measurement Challenge by stefano.palestini http://www.hubbardresearch.com/2010/03/the-measurement-challenge/comment-page-1/#comment-300 stefano.palestini Thu, 01 Dec 2011 14:26:14 +0000 http://blog.hubbardresearch.com/?p=491#comment-300 Douglas, We are improving new procedure for financial planning whose target is to reduce the range of approximation of the foreseen from the actual range +- 20 Million likelihood 30% (as the greatest difference between actual and forecast of net financial position) +- 5 Million likelihood 70% (the most narrow difference actual vs forecast) , to the new interval +-15M E 10% /+- 5M 80%. The cost of the investment is Euro 500 K. It is possible and meaningful measure the value of the better information available in the new procedure of financial planning using EVPI as described in the capitol “Measuring Value of Information” in your book How to Measure Anything (Paragraph “the value of information for ranges)? Else what you suggest to set better the problem of the value of information? Thank you in advance for your answer. stefano palestini internal auditing & risk management Douglas,
We are improving new procedure for financial planning whose target is to reduce the range of approximation of the foreseen from the actual range +- 20 Million likelihood 30% (as the greatest difference between actual and forecast of net financial position) +- 5 Million likelihood 70% (the most narrow difference actual vs forecast) , to the new interval +-15M E 10% /+- 5M 80%. The cost of the investment is Euro 500 K.
It is possible and meaningful measure the value of the better information available in the new procedure of financial planning using EVPI as described in the capitol “Measuring Value of Information” in your book How to Measure Anything (Paragraph “the value of information for ranges)?
Else what you suggest to set better the problem of the value of information?
Thank you in advance for your answer.

stefano palestini
internal auditing & risk management

]]> Comment on Pulse Downloads(Registration Required) by Pulse: The New Science http://www.hubbardresearch.com/pulse-downloads/comment-page-1/#comment-299 Pulse: The New Science Fri, 04 Nov 2011 02:44:27 +0000 http://blog.hubbardresearch.com/?page_id=615#comment-299 [...] Reader Downloads [...] [...] Reader Downloads [...]

]]> Comment on Pulse Downloads(Registration Required) by Pulse: The New Science | Pulse: The New Science http://www.hubbardresearch.com/pulse-downloads/comment-page-1/#comment-298 Pulse: The New Science | Pulse: The New Science Thu, 03 Nov 2011 22:41:08 +0000 http://blog.hubbardresearch.com/?page_id=615#comment-298 [...] Reader Downloads [...] [...] Reader Downloads [...]

]]> Comment on Evolving Taxonomy of Major Risks by Milewskp http://www.hubbardresearch.com/2009/09/evolving-taxonomy-of-major-risks/comment-page-1/#comment-296 Milewskp Wed, 26 Oct 2011 16:49:41 +0000 http://blog.hubbardresearch.com/?p=345#comment-296 Hi Doug, I've read somewhere (likely in your book) that people find it difficult to estimate very low probability events, and that they are inherently bad at it. The theory is that this has evolutionary roots: If cave man Joe didn't know of anyone who was attacked by a sabre-tooth tiger, he didn't spend a lot of time pondering the odds. He was better off spending his time mitigating the real risks to his survival. So, since there was never a genetic need to be good at estimating low probability events, our brains never developed the skill. Consequently, unless they have the data, it is difficult for the average person to estimate the probability of their house burning down, totalling their car, or dying in a plane crash. And if they tried, their estimate would likely be inaccurate, maybe off by orders of magnitude. Strangely, we do seem to be more comfortable estimating risk (i.e., Consequence x Probability) directly; eg., if I would be willing to pay up to $X for auto, home or travel insurance, it means I estimate the risk to be X. But, does more comfort mean that the estimate is better? Hi Doug,
I’ve read somewhere (likely in your book) that people find it difficult to estimate very low probability events, and that they are inherently bad at it. The theory is that this has evolutionary roots: If cave man Joe didn’t know of anyone who was attacked by a sabre-tooth tiger, he didn’t spend a lot of time pondering the odds. He was better off spending his time mitigating the real risks to his survival. So, since there was never a genetic need to be good at estimating low probability events, our brains never developed the skill.
Consequently, unless they have the data, it is difficult for the average person to estimate the probability of their house burning down, totalling their car, or dying in a plane crash. And if they tried, their estimate would likely be inaccurate, maybe off by orders of magnitude.
Strangely, we do seem to be more comfortable estimating risk (i.e., Consequence x Probability) directly; eg., if I would be willing to pay up to $X for auto, home or travel insurance, it means I estimate the risk to be X.
But, does more comfort mean that the estimate is better?

]]> Comment on Still no evidence… by spoppe2001 http://www.hubbardresearch.com/2010/02/483/comment-page-1/#comment-294 spoppe2001 Sun, 25 Sep 2011 02:07:43 +0000 http://blog.hubbardresearch.com/?p=483#comment-294 Just finished FRM. I am immersed daily in innumerable IT risk assessment questionnaires, which are now routinely hundreds of questions long and getting longer all the time. The relevance of some questions to the assessment of risk is hard to fathom. As a supplier of IT services we have a finite budget for reducing risk for customers. I hope to use the ideas in FRM to build an objective basis for deciding whether to invest further in a particular area. Comments from you and your readers who have made progress in this area are most welcome. Just finished FRM. I am immersed daily in innumerable IT risk assessment questionnaires, which are now routinely hundreds of questions long and getting longer all the time. The relevance of some questions to the assessment of risk is hard to fathom. As a supplier of IT services we have a finite budget for reducing risk for customers. I hope to use the ideas in FRM to build an objective basis for deciding whether to invest further in a particular area. Comments from you and your readers who have made progress in this area are most welcome.

]]> Comment on How To Measure Anything Downloads for 1st and 2nd Editions(Registration Required) by chaimr http://www.hubbardresearch.com/htma-downloads/comment-page-1/#comment-293 chaimr Wed, 21 Sep 2011 07:43:44 +0000 http://blog.hubbardresearch.com/?page_id=44#comment-293 Dear Doug, You state in the description for the download to chapter 10 for the Bayesian Inversion for Population Proportion Range (Exhibit 10.4) that it is for the first edition. I have the second edition and was unsure about the statement in the [spreadsheet]: "ERRATA: There was an error in the book example. This spreadsheet correct that error." Did this statement also apply to the 2nd edition, or was the error already corrected? I think you should make it clear in the [spreadsheet] itself. Sincerly, Chaim Dear Doug,

You state in the description for the download to chapter 10 for the Bayesian Inversion for Population Proportion Range (Exhibit 10.4) that it is for the first edition.
I have the second edition and was unsure about the statement in the [spreadsheet]:
“ERRATA: There was an error in the book example. This spreadsheet correct that error.”

Did this statement also apply to the 2nd edition, or was the error already corrected? I think you should make it clear in the [spreadsheet] itself.

Sincerly,
Chaim

]]> Comment on Quote of the Day – 08.10.2011 by dwhubbard http://www.hubbardresearch.com/2011/08/quote-of-the-day-08-10-2011/comment-page-1/#comment-291 dwhubbard Thu, 25 Aug 2011 14:35:16 +0000 http://www.hubbardresearch.com/?p=1527#comment-291 I do mention the "you can't manage what you can't measure" quote but I don't assert that all management without measurement is ineffectual. I would, however, argue that the facts show that a wide variety of decisions are easily improved when augmented by measurements - including the decision about which methods are better. What I get in to later in the book is some of the research that shows how flawed our judgment can be when we rely on inution. These are based on the findings of multiple researchers each using a large number of subjects. When we see the kinds of errors which even seasoned experts include in subjective judgment, alternatives have to be considered. I don't recall exactly what I said about an actuaries opinion being better than subjective managment judgment in the beginning of the book, but I assure you I only state what the evidence allows me to state. In later chapters I describe evidence that shows how frequent certain kinds of errors are in subjective judgment. Experts routinely make incorrect inferences from data that would easily be corrected with simple calculations (see my analysis of the NASA study done by Robin Dillon and the work of Paul Meehl) and experts that rely more on quantitative methods measurably outperform experts who do not (the Fiona Lamb study of oil companies and my analysis of another set of NASA data). But we also know that subjective methods can be useful especially when experts are trained to provide subjective estimates of actual quantities (not "high, medium, low") and that if certain errors and inconsistencies are controlled for (such as I mention in the Lens method) then expert estimates can be improved. We know that without these certain kinds of training and adjustments experts in all fields are systemically overconfident and highly inconsistent in their judgments. Even the error added by popular ordinal scoring methods has been measured (Budescu, Cox and others I mention). You will see all of those parts in the book. V/R Doug I do mention the “you can’t manage what you can’t measure” quote but I don’t assert that all management without measurement is ineffectual. I would, however, argue that the facts show that a wide variety of decisions are easily improved when augmented by measurements – including the decision about which methods are better. What I get in to later in the book is some of the research that shows how flawed our judgment can be when we rely on inution. These are based on the findings of multiple researchers each using a large number of subjects. When we see the kinds of errors which even seasoned experts include in subjective judgment, alternatives have to be considered.

I don’t recall exactly what I said about an actuaries opinion being better than subjective managment judgment in the beginning of the book, but I assure you I only state what the evidence allows me to state. In later chapters I describe evidence that shows how frequent certain kinds of errors are in subjective judgment. Experts routinely make incorrect inferences from data that would easily be corrected with simple calculations (see my analysis of the NASA study done by Robin Dillon and the work of Paul Meehl) and experts that rely more on quantitative methods measurably outperform experts who do not (the Fiona Lamb study of oil companies and my analysis of another set of NASA data). But we also know that subjective methods can be useful especially when experts are trained to provide subjective estimates of actual quantities (not “high, medium, low”) and that if certain errors and inconsistencies are controlled for (such as I mention in the Lens method) then expert estimates can be improved. We know that without these certain kinds of training and adjustments experts in all fields are systemically overconfident and highly inconsistent in their judgments. Even the error added by popular ordinal scoring methods has been measured (Budescu, Cox and others I mention). You will see all of those parts in the book.

V/R
Doug

]]> Comment on Quote of the Day – 08.10.2011 by thefrasers http://www.hubbardresearch.com/2011/08/quote-of-the-day-08-10-2011/comment-page-1/#comment-290 thefrasers Thu, 25 Aug 2011 14:11:40 +0000 http://www.hubbardresearch.com/?p=1527#comment-290 Thanks for the reply; I apologise because I am definitely jumping the gun by commenting when I have only just begun the book. I recognize that I'm in danger of sounding like a politician who complains about a film they haven't watched. The "all models are wrong, but some are useful" quote is a favourite of mine because it seems to emphasise the importance of judgement. I was also impressed with your initial question along the lines of "How do you know it works?" as this is indeed the (multi-)million-dollar question. Difficult to answer as the fact that I've never been attacked by a tiger doesn't mean that my "anti-tiger attack" blue shirt is necessarily a reasonable mode of defence, irrespective of any number of historical measurements. I need to read more - I'm not attempting to set up a straw man although I may be doing so unintentionally. I just have an innate suspicion of the "if it can't be measured it can't be managed/judged" style of thinking. I am not claiming that your book follows this approach because I don't have enough experience of the book yet! I was just unnerved at the beginning where I recall a passage which seemed to suggest that the approval of an actuary would be better than using subjective management judgement. Note that this may well be a superficial mis-remembering, and I apologise. Perhaps better that I read rather than comment. Many thanks, Peter Fraser Thanks for the reply; I apologise because I am definitely jumping the gun by commenting when I have only just begun the book. I recognize that I’m in danger of sounding like a politician who complains about a film they haven’t watched. The “all models are wrong, but some are useful” quote is a favourite of mine because it seems to emphasise the importance of judgement. I was also impressed with your initial question along the lines of “How do you know it works?” as this is indeed the (multi-)million-dollar question. Difficult to answer as the fact that I’ve never been attacked by a tiger doesn’t mean that my “anti-tiger attack” blue shirt is necessarily a reasonable mode of defence, irrespective of any number of historical measurements.
I need to read more – I’m not attempting to set up a straw man although I may be doing so unintentionally. I just have an innate suspicion of the “if it can’t be measured it can’t be managed/judged” style of thinking. I am not claiming that your book follows this approach because I don’t have enough experience of the book yet! I was just unnerved at the beginning where I recall a passage which seemed to suggest that the approval of an actuary would be better than using subjective management judgement. Note that this may well be a superficial mis-remembering, and I apologise.

Perhaps better that I read rather than comment.

Many thanks,
Peter Fraser

]]>