(630) 858-2788

Learn more about Hubbard Decision Research

Find and download content referenced in our Publications

Downloads and VIP Subscriber Library

Originally posted on http://www.howtomeasureanything.com/forums/ on Monday, June 22, 2009 4:33:03 AM.

“Greetings!

ON the page 187 there is a claim that the author has generated an example of random power law generator, but i cant find it from the examples. Can someone help me with this problem who has find the example from downloads?

THanks,

Markus Kantor”

Thanks for your question. I had it up briefly and discovered a flaw in the automatic histogram generation. I’m out of the country I’ll have the power law generator up by the end of June (much sooner if I find the time before I head back to the US).

Thanks for your patience

This question was originally posted on Friday, June 12, 2009 10:47:57 AM by mzaret20000 on http://www.howtomeasureanything.com/forums/.

“Hi Douglas,

I have been asked to provide the gross margin for client engagements. My company is a recruiting firm that operates like an internal recruiting team (meaning that we charge fixed rates as we are doing the work regardless of outcome rather than billing a percentage of compensation hired). Most of our engagements are small so they aren’t not profitable in the absence of other engagements. I’m wondering what your strategy would be to make this type of measurement.”

Thanks for your post. At first glance, your problem seems like it could just be a matter of accounting procedures (e.g. revenue minus expenses and divided by revenue with consideration for issues like how you allocate marketing costs across projects, etc.) but let me presume you might mean something that is more complex. It might not be what I strictly call a “measurement” since it sounds like you probably have the accounting data you need already and you probably do not actually need to make additional observations to calculate it. This is more of a calculation based on given data and the issue is more about what it is you really want to compute.

Since you mentioned that your projects are small and not profitable n the absence of other engagements, perhaps what you really want to compute is some kind of break-even point based on fixed costs and marginal costs of doing business. But even that’s a guess on my part.

Perhaps you could describe why you need to know this. This is a typical question I ask. What decision could be different given this information? What actions of what parties would this information guide? Once you define that I find that the measurement problem is generally much clearer.

Originally posted on http://www.howtomeasureanything.com/forums/ on Wednesday, July 08, 2009 2:46:05 PM by

“Hi Doug,

I want to share an observation a V.P. made after doing the 10 pass fail questions. If one was to input 50% confidence to all the questions and randomly selected T/F they would be correct 1/2 the time the difference would be 2.5.

The scoring would indicate that that person was probably overconfident. Can you help here ?.

I am considering making the difference between the overall series of answers (as a decimal) and the Correct answers(as a decimal) as needing to be greater than 2.5 for someone to be probably overconfident.

please advise

Thaks in advance – Hugh”

Yes, that is a way to “game the system” and the simple scoring method I show would indicate the person was well calibrated (but not very informed about the topic of the questions). It is also possible to game the 90% CI questions by simply creating absurdly large ranges for 90% of the questions and ranges we know to be wrong for 10% of them. That way, they would always get 90% of the answers within their ranges.

If the test-takers were, say, students, who simply wanted to appear calibrated for the purpose of a grade, then I would not be surprised if they tried to game the system this way. But we assume that most people who want to get calibrated realize they are developing a skill they will need to apply in the real world. In such cases they know they really aren’t helping themselves by doing anything other than putting their best calibrated estimates on each individual question.

However, there are also ways to counter system-gaming even in situations where the test taker has no motivation whatsoever to actually learn how to apply probabilities realistically. In the next edition of How to Measure Anything I will discuss methods like the “Brier Score” which would penalize anyone who simply flipped a coin on each true/false question and answered them all as 50% confident. In a Brier Score, the test taker would have gotten a higher score if they put higher probabilities on questions they thought they had a good chance of getting right. Simply flipping a coin to answer all the questions on a T/F test and calling them each 50% confident produces a Brier score of zero.

Thanks for your interest,

Doug Hubbard

SiteLock