The Measurement Challenge

Steven,

Thanks for your input. I would argue that whether a measurement is “direct” or not is an unambiguous distinction mathematically, semantically and at the level of fundemental epistemology. This distinction adds no clarity whatsoever because most – or perhaps all – measures of properties in physical sciences are not “direct”. They rely on readings of an instrument (e.g. an digital scale, ohmmeter or photometer) and rely on inferences from related observations (e.g. observing the deflection of a particle in a magnetic field or the movement of planet to measure mass). I would even argue that we only percieve reality indirectly in the first place. All we can do is make observations to make inferences that ultimately have some utility to practical decisions.

In the definition I propose for measurement (which is, in fact, consistent with information theory, decision theory and measurement theory) income really is a a measure of IQ which is, in turn, a measure of intelligence. Of course, this would be a measure with a huge amount of error but, on average, estimates based on income would be slightly more accurate than estimates based on no information at all. If you doubt this, let me propose a game that would test whether you really believe your position. Let’s identify 100 people who have had IQ tests representing a wide range of IQ’s. We will know nothing about them and we will not be told their IQ’s, except that I will be told thier income from last year and you will not. We will sort them into random pairs and we will bet which person is smarter. You should be indifferent because you would have no information about whether one person has a higher IQ. But since I know their incomes, I will not be indifferent. I will, for each pair, choose that the person with the higher income also has the higher IQ. Each time I’m wrong I pay you $100 and each time I win you pay me $100. I would love to play this game for a million people if we could. If I’m right only slightly more often then I’m wrong, then I would make a lot of money off of you. I would even be willing to pay you $500 to play this game with me for 100 persons. Now would you prefer to be the one of us who had the income information? Would you even be willing to pay me if you had it? You might say you are not a betting person, but I think betting is simply the ultimate test of whether a person really believes some position they have. You might say this is impractical but I think we could find a way to make it happen.

The fact is that even when there is a small correlation between X and Y, knowledge of X slightly reduces your uncertainty about Y. And I would say that any definition of measurement that you propose that ignores the reduction in uncertainty would be far more confusing. You would have to resort to a definition that merely describes a procedure regardless of whether the result is informative. And if you said that the procedure only resulted in a measurement if it was informative while rejecting that the threshold is any uncertainty reduction, then you would have to define how much uncertainty reduction is required. Presumably, you would have to define this arbitrary threshold differently for every possible kind of measurement since error in measurements vary widely among fields.

Taking your claim “Clearly, to say Income is a measure of anything other than Income makes the concept of measure unnecessarily confusing and much more prone to be misapplied.” to its logical conclusion, you would have to reject the validity of most measures in the physical sciences since, as I said, most are inferences based on indirect observations (nobody has ever “directly” measured the mass of an electron or a star or the age of a rock or even time itself). If, as you say “IQ is only a measure of IQ” appears to say that if X is correlated to Y, X is still only a measure of X and it reveals nothing about the possible values of Y. If this is the case, then you must also hold that the glow of a hot body is a measure only of its glow, not its temperature. You must hold that the depth of a rock formation is only a measure of its depth, not its age. You must hold that a credit score is only a measure of a credit score and says absolutely nothing about whether someone is an acceptable lending risk – meaning that a 1000 people with credit scores of under 600 are just as good a credit risk as 1000 people with scores over 750. There will be exceptions, of course, but given a portfolio of 1000 people, do you really want to charge the same interest for the under-600 score group as the over-750 score group?

You stated “The main policy problem with substituting IQ for Intelligence is not so much that the correlation is not 100%, but rather than the correlation seems to vary quite a bit among different populations. In so many cases, the method for deriving a concrete measure that correlates well with an intangible concept is very context sensitive. For example, deriving a concrete measure that correlates well with intelligence in western populations may not correlate well in African populations.” How is this more uniquely “context specific” than any other measure I just mentioned? As before, if this context-specificity you speak of was any real obstacle at all to measurement, then again, most of what we know from science could not be possible. There is no complexity you can think of that does not have a measurement solution. Many of the procedures in scientific method are specifically to controll for such issues.

Or perhaps you believe that it is not the “directness” but the mere existance of error that undermines any measurement. There is a common fallacy that the the existance of noise means a lack of signal. This would force you to defend yet another position that would be impossible to reconcile with all scientific knowledge. Or perhaps you believe that the existance of noise (i.e. error) in a measurement means that any signal that does exist must not have any utility. This would be impossible to reconcile with both decision theory and common sense. If you knew a coin favored heads on a flip just 52% of the time, that knowledge would be worth a lot of money over a large number of flips.

On the other hand, I don’t deny that people misuse measures. Regarding your comment about productivity measures, if someone actually equates a measure of productivity on a factory floor to productivity in creating software, I could only respond that this would be a straw man argument. I make no claim that two obviously uncorrelated factors say anything about each other. I also point out in How to Measure Anything that most organizations measure the wrong things (because they are not computing the value of uncertainty reduction). If they applied this method consistently, they would measure what matters. And, as a side note, don’t confuse measures used for decisions like project approval to incentive programs. They are two different issues. Good incentives are based on good measures but that doesn’t mean any measurement should be part of some incentive. I clearly argue against this in the book.

And don’t forget that you always are comparing a decision analysis method based on measurements to some *other* method – presumably your intuition. That also has a measurable performance and research shows it is often not hard to beat with even simple quantitative methods.
Again, if you really believe what you believe, then let’s start recruiting some people for our bet.

Thanks for your contribution to the conversation and I look forward to your response.

Doug Hubbard

Stefano,

This sounds like a perfectly viable problem for the value of information. However, I would need clarification on something you said. It seems you are saying that the current accuracy of forecasts to actuals has a 30% chance of being within 20 million and a 70% chance of being within 5 million, right? That seems backward to me. The wider range should have the higher probability unless I misunderstand how you are stating this. The same would seem to hold for the target accuracy.

But, that aside, improving the value of forecasts is certainly something the information value pertains to. You have a cost of overestimating and/or a cost of underestimating, right? How much would you lose for every million you over or under estimate? The product of this “loss function” and your probability distribution for the forecast is the expected opportunity loss (EOL). You have an EOL for the current state and for the desired target accuracy. The difference between the two EOLs is the value of information. Technically, you would only be computing EVPI if you were comparing the value of your current accuracy to perfect forecasts, which I’m sure is not what you mean.

Thanks for your comment,
Doug Hubbard