Chapter 6

Originally posted on on Thursday, February 19, 2009 1:41:48 PM, by Thakur.

“I enjoyed reading Chapter 6 (Measuring Risk: Introduction to the Monte Carlo Simulation). It was very informative. After reading it I tried to do the following using Excel. But I failed.

1). Simulating the Monty Hall Problem.

2). Simulating Birthdays

3). Genetics: Simulating Population Control

Can You please help me and guide me.



You are asking for a lot! But how about I answer a bit at a time? First, lets do Monty Hall.

For those of you who might not have heard of this problem, its based on a classic probability theory example. Imagine that you are on the 70’s game show “Let’s Make a Deal” hosted by Monty Hall. You are a contestant and you are given three doors to choose from. Behind one of the doors is a brand new car! If you choose the door with the car behind it, you get to drive it away.

You choose a door. But then Monty Hall shows you what is behind one of the other doors to reveal one of the “joke prizes” (e.g. a donkey). Then he asks you if you would like to keep the door you first chose or switch to the other remaining door. People often think that the odds of winning would be the same whether you switched or not. But they would be wrong.

To demonstrate why switching doors would be better, let’s set up a spreadsheet simulation where we define columns for the prize door, the chosen door, and the revealed door. One more column will be used as a flag to indicate whether we would have won if we stayed with the first door we chose or if we should have switched doors. Then we will copy down the first row of these columns to a few thousand rows to see the outcome.

Column 1, The Prize Door: This is the door the prize is really behind. As a contestant, you wouldn’t know this information, but we need it for the simulation. Write “The Prize Door” in cell A1. In cell A2 write =int(rand()*3+1). This will randomly generate the value of 1, 2 or 3.

Column 2, The Chosen Door: This is the door the contestant chose. In B1, write “The Chosen Door” and in B2 write the same formula you wrote in A2; =int(rand()*3+1). Again, this will randomly generate the value of 1, 2 or 3.

Column 3, The Revealed Door: This is the door Monty Hall reveals. Monty will always reveal a door you didn’t choose and it will always be a door that doesn’t have a prize behind it. In cell C1 write “The Revealed Door” and in C2 write =if(and(a2=1,b2=1),int(rand()*2+2),if(and(a2=1,b2=2),3,if(and(a2=1,b2=3),2,if(and(a2=2,b2=1),3,if(and(a2=2,b2=2),int(rand()+.5)*2+1,if(and(a2=2,b2=3),1,if(and(a2=3,b2=1),2,if(and(a2=3,b2=2),1,int(rand()*2+1))))))))) This seems clumsy, but its visually easier to decompose and understand than some approaches I might have taken. This will generate values according to the following table:

Prize Door……Chosen Door……Revealed Door
1…………………..1………………….2 or 3
2…………………..2………………….1 or 3
3…………………..3………………….1 or 2

Column 4, Winning Strategy; This cell tells you what the winning strategy would have been. Either you stick with the door you first chose or you switch doors. In D1 write “Winning Strategy” and in D2 write =if(A2=b2,0,1). This will generate a 0 if the winning strategy would have been to stick with the door you have and a 1 if you were better off switching.

Now copy down row 2 a thousand rows and take the average of the values in column 4 (remember not to average in the text in D1). One way to do this is write =average(D2:D1001) in cell E1. If you were just as well off sticking with the first chosen door as switching, then this average would be .5. But you will find that the average will be about .667. In other words, two thirds of the time the winning strategy was switching doors. The reason this works is that when Monty Hall reveals one of the other doors, he gives you additional information you didn’t have before. He reveals ONLY a door that doesn’t have a prize and ONLY a door you didn’t choose. When you first choose a door, you have a 2/3 chance of not winning (the prize is behind one of the other two doors). Once he reveals which of the other 2 doors is not a winner, then the remaining door has a 2/3 chance of winning.

Check back for my responses to your other questions. For clarification, when you talk about birthdays do you mean simulating the problem where you find minimum number of people before there is equal odds that at least 2 people have the same birthday?

Thanks for your question
Doug Hubbard

Value of Foresight

Originally posted on on Wednesday, February 18, 2009 4:54:25 PM, by RossShott.

“Hello Mr. Hubbard,

Your book “How to Measure Anything” is remarkable!

I am currently completing my Master of Technology in Futures Studies degree by working on my thesis project. The project entails fleshing out a Foresight Maturity Model (similar to a CMM or CMMI) recently developed by Social Technologies. Meaning that I have been given the task of figuring out how to measure foresight within companies in such a way that those companies can be evaluated by a Futurist consulting firm using the FMM tool or the company could apply the FMM tool to their own organization for foresight improvement efforts. How do you measure foresight capability I wondered? I wondered even more about how do you measure the value of improved foresight? (By foresight I mean having a better ability to forecast alternative futures for robust decision-making and putting in place practices which will enable you to better prepare for changes and unexpected events in the future).

I searched the library and the internet before purchasing a handful of books on measurement and experiment design. Your book is a diamond among a sea of broken glass. “How to Measure Anything” is so well written I could barely put it down. I read it straight through (including doing the calibration exercises) in two nights after work and began reading it a second time with highlighter in hand.

I can’t thank you enough for the incredible insights and inspiration you have given me! Your book with go to the top of the stack as I begin my Ph.D. in this Fall.

Best regards,

Ross Shott
Graduate Student, University of Houston”


Thanks for the kind endorsement and thanks for bringing this conversation over from the email conversation that we started.

My first question for you is the same one with which I begin all measurement problems: “What do you mean by foresight, exactly?” If you can think about this question and those below, I’ll get closer to solving your problem.

But while you consider that, let me propose a possible meaning and you can modify it further. I suppose you might mean the ability to predict particular events. Specifically, foresight might have more to do with forecasting trends like “What is the next big bio-tech revolution?” instead of “Will it rain tomorrow?”. But correct me if you mean something else.

For now, lets suppose you mean what I suggested and consider what its value might be. Presumably, foresight has value because of the effect it would have on your decisions and behaviors. As I ask in the book, what would you imagine you would be doing differently if you had more foresight? Imagine a list of specific events and ask what would be the value of acting on it.

Thinking about these specific events will help us focus on the right question. Tell me as much as you can about this application and let generate some specific examples of “improved foresight” and, then, some specific examples of what you would be able to do better if you had that foresight. Once we have that, let’s follow the path as shown in the book:

1) Decompose it – Think of a list of specific events you would have foresight of, a list of specific actions you might take if you had such foresight, and the specific advantages of those actions. Think of how all these might come together into a formula that computes the value of foresight.
2) State your current uncertainty – for each of the decomposed items in step #1, state what you know now about it. Estimate how often these events might occur. Estimate the possible actions and estimate the impacts of good and bad outcomes.
3) Compute the value of information for each of the variables – this requires that you think of the costs and benefits of some particular alternative you might take differently if you had more information about the value of foresight. Are you considering a specific software purchase that would allegedly improve foresight? Then that is the decision this information would help with.
4) Design the empirical method – for the variable(s) with the highest information value, think about how you would observe it. Then, based on the possible observation, design a systematic observation based, say, on random sampling or controlled experiments.

This sounds rather vague right now, but we can get more specific if you provide more specifics about the problem.


Doug Hubbard

Books Related to Calibration

Originally posted on on Monday, February 16, 2009 11:32:17 AM.

“I am looking for some material (articles or books) on the subject of Calibration. I want to be expert in Calibration.


I certainly support your goal for becoming and expert in this topic. It is a well-studied topic but is still far too obscure in practical applications. Beyond my book, the most important sources are the purely academic literature…which I would definitely recommend for anyone who wants to be an expert. My next book The Failure of Risk Management, will cover this topic with a slightly different emphasis and, in some cases, in more detail. In both books, I resort to several academic studies, including the following.

A key source is Judgment Under Uncertainty: Heuristics and Biases, Cambridge University Press, 1982. It is a compilation of several research papers on the topic. You can buy this book on Amazon.

Here are several more articles:

A.H. Murphy and R. L. Winker, ‘‘Can Weather Forecasters Formulate Reliable Probability Forecasts of Precipitation and Temperature?,’’ National Weather Digest 2, 1977, 2–9.

D. Kahneman and A. Tversky, ‘‘Subjective Probability: A Judgment of Representativeness,’’ Cognitive Psychology 3, 1972, 430–454.

G.S. Tune, ‘‘Response Preferences: A Review of Some Relevant Literature,’’ Psychological Bulletin 61, 1964, 286–302.

E. Johnson, ‘‘Framing, Probability Distortions and Insurance Decisions,’’ Journal of Risk and Uncertainty 7, 1993, 35.

D. Kahneman and A. Tversky, ‘‘Subjective Probability: A Judgment of Representativeness,’’ Cognitive Psychology 4, 1972, 430–454.

D. Kahneman and A. Tversky, ‘‘On the Psychology of Prediction,’’ Psychological Review 80, 1973, 237–251.

A. Tversky and D. Kahneman, ‘‘The Belief in the ‘Law of Small Numbers,’’’ Psychological Bulletin, 1971.

A. Koriat, S. Lichtenstein, and B. Fischhoff, ‘‘Reasons for Confidence,’’ Journal of Experimental Psychology: Human Learning and Memory 6, 1980, 107–118

I am concerned about the CI, median and normal distribution

Originally posted at, on Wednesday, February 11, 2009 2:16:38 PM, by andrey.

“Hello Douglas,

First of all let me say I have thoroughly enjoyed reading your book. I have a technical background (software engineering) and have always been surprised at how “irresponsible” some business-level decision making can be – based on gut instincts and who-knows-what. This ‘intuitive’ approach is plagued with biases and heuristics, the effects of such approach has been widely publicized (for example here). This is one of many reasons I found your book very simulating and the AIE approach as a whole very promising.

However I have reservations about a few points you make. Please forgive me my ignorance if my questions are silly, my math has become rusty with the years passing by.

One of my concerns is the validity of the assumption that you make when explaining ‘The Rule of Five’, 90% CI and especially when using Monte Carlo simulation. I can believe (although it would’ve been great to see the sources) that ‘there is a 93% chance that the median of a population is between the smallest and largest values in any random sample of five from that population’. But when you are applying this to the Monte Carlo simulations, you assume that the mean (which is also the median for symmetric probability distributions) is exactly in the middle of the confidence interval. Which, I think, makes a big difference to the outcome because of the shape of the normal distribution function. If you assume the the median is for example very close to the lower or upper bound of the confidence interval by putting a different value into the =norminv(rand(),A, B) formula the results would be different.

I am still working through your book (second reading), trying to ‘digest’ and internalise it properly. I would be very grateful if you could explain this to me.

Thank you very much,


Thanks for your comment.

I don’t show a source (I’m the one who coined the term “Rule of Five” in this context) but I show the simple calculation, which is easily verifiable. The chance of randomly picking one sample with a parameter value above the true population median for that parameter is, by definition, 50%. We ask “what is the probability that I could happen to randomly choose five samples in a row that are all above the true population median?” It is the same chance as flipping five coins and getting all heads. The answer is 1/2^5 or 3.125% Likewise, the probability that we could have just picked 5 in a row that were all below the true population median is 3.125%. That means that there is a 93.75% chance that some were above and some were below – in other words, that the median is really between the min and max values in the sample. It’s not an “assumption” at all – it is a logically necessary conclusion from the meaning of the word “median”.

You can verify this experimentally as well. Try generating any large set of continuous values you like using any distribution you like (or just define a distribution function for such a set). Determine the median for the set. Then randomly select 5 from the set and see if the known population median is between the min and max values of those 5 samples. Repeat this a large number of times. You will find that 93.75% of the time the known median will be between the min and max values of the sample of 5.

I believe I also made it clear that this only applies to the median and not the mean. I further stated that if, on the other hand, you were able to make the determination that the distribution is symmetrical then, of course, it applies to the mean as well. Often, you may have reason to do this and this is no different than the assumption in any application of a t-stat or z-stat based calculation of a CI (which are always assumed to be symmetrical).

Furthermore, you certainly should not use a function that generates a normally distributed random number if you know it not to be normally distributed and I don’t believe I recommended otherwise. If you know the median and the mean of the distribution you intend to generate are not the same, then you can’t count on Excel’s normdist function to be a good approximation. For the sake of simplicity, I gave a very limited set of distributions functions for random values in Excel. But we can certainly add a lot more (triangular, beta, lognormal, etc.)

My approach is not to assume anything you don’t have to. If you don’t know that the distribution isn’t lopsided, you can simulate that uncertainty, too. Is it possible that the real distribution could actually be lognormal? Then put a probability on that and generate accordingly. Why “assume” something is true if we can explicitly model that we are uncertain about something?



In Russian

Hello, [I have] short question – where i can find the book in Russian? On the site it is written that the translation exists.

I`m writing postgraduate research on the efficiency of IT – it is quite interesting to take a deep look into your methodology, but the translation can significantly increase the speed of this process))

[Thanks] in advance,