How do you backtest a probability estimate?

by | Aug 25, 2009 | News, Reality Check on Methods

A question asked by a subscriber:

I am sympathetic to the concept of using a Monte Carlo approach, perhaps using a cascaded series of Monte Carlo models to model inputs, to produce an estimated probability distribution. This seems like a more reasonable approach to modeling reality than to assume the distribution follows a normal or lognormal distribution.

To me, this seems like a logical extension of the budgeting approach familiar to every accountant, with incremental improvements to such a model building through time. So long as someone does not become ‘wedded to the model’, this process is powerful.

To avoid becoming ‘wedded to the model’, it seems to me that it is necessary to identify the parameters for your inputs (or the environment which affects those inputs) within which you believe your model will be robust. Movements of your inputs outside of this range should trigger a re-evaluation of your model.

For those who are wedded to VaR, you can even ‘measure’ the risk associated with your model as a 5% VaR etc. if you want to lose much of the detail of what you have done .. there is sometimes a place for a summary measure, so long as it does not become the input to a subsequent calculation.

I am convinced of the importance of accurate calibration and backtesting of models.

What I am less clear about is how you can ‘backtest’ a probabilistic model on anything other than a gross basis. How do you know whether an observed event is a 1 in a 100 event or a 1 in 10 event? Clearly if there is sufficient data, then the usual probabilistic maths can be used .. but what about where we are dealing with unusual, but perhaps critical, events?

Is the only answer to use traditional statistics to measure the confidence we have in our model? And if so, how can these measures be incorporated into the re-iteration of our Monte Carlo model?

First Print Errata

by | Aug 25, 2009 | Errata, How To Measure Anything Blogs, News

Welcome to the Errata thread in the book discussion of the How to Measure Anything Forum. An author goes through a lot of check with the publisher but some errors manage to get through. Some my fault, some caused by the typesetter or publisher not making previous changes. I just got my author’s copies 2 days ago (July 20th) about 2 weeks before it gets to the stores. But I already found a couple of errors. None should be confusing to the reader, but they were exasperating to me. Here is the list so far.

1) Dedication: My oldest son’s name is Evan, not Even. My children are mentioned in the dedication and this one caused by wife to gasp when she saw it. I don’t know how this one slipped through any of the proofing by me but this is a big change priority for the next print run.

2) Preface, page XII: The sentence “Statistics and quantitative methods courses were still fresh in my mind and I in some cases when someone called something “immeasurable”; I would remember a specific example where it was actually measured.” The first “I” is unnecessary.

3) Acknowledgements, page XV: Freeman Dyson’s name is spelled wrong. Yes, this is the famous physicist. Fortunately, his name is at least spelled correctly in chapter 13, where I briefly refer to my interview with him. Unfortunately, the incorrect spelling also seems to have made it to the index.

4) Chapter 2, page 13: Emily Rosa’s experiment had a total of 28 therapists in her sample, not 21.

5) Chapter 3, Page 28. In the Rule of Five example the samples are 30, 60, 45, 80, and 60 minutes so the range should be 30 to 80- not 35 to 80.

6) Chapter 7: Page 91, Exhibit 7.3: In my writing, I had a habit of typing “*“ for multiplication since that is how it is used in Excel and most other spreadsheets. My instructions to my editor were to replace the asterisks with proper multiplication signs. They changed most of them throughout the book but the bottom of page 91 has several asterisks that were never changed to multiplication signs. Also, in step 4 there are asterisks next to the multiplication signs. This hasn’t seemed to confuse anyone I asked. People still correctly think the values are just being multiplied but might think the asterisk refers to a footnote (which it does not).

7) Chapter 10: Page 177-178: There is a error in the lower bound of a 90% confidence interval at the bottom of page 177 . I say that the range is “79% to 85%”. Actually, the 79% is the median of the range and the proper lower bound is 73%. On the next page I show an error in the column headings of Exhibit 10.4. I say that the second column is computed by subtracting one normdist() function in Excel from another. Actually, the order should be reversed so that the first term is subtracted from the first. As it is now, the formula would give a negative answer. Taking the negative of that number gives the correct value. I don’t think this should confuse most readers unless they try to recreate the detailed table (which I don’t expect most to do). Fortunately, the downloadable example spreadsheet referred to in this part of the book corrects that error. The correction is in the spreadsheet named Bayesian Inversion, Chapter 10 available in the downloads.

8) Chapter 12: page 205; There should be a period at the end of the first paragraph.