The Problem of Heterogeneity
"About Schmidt" is a nice, sad and funny show about a retired actuary. For this movie, Jack Nicholson won the Best Actor title in the Golden Globe Awards. I came across this in Warren Schmidt’s letter to his foster child Ndugu,
"I believe I mentioned in my previous letter that I was an actuary at Woodmen and The World insurance company. If I am given a man’s age, race, profession, place of residence, marital status and medical history, I can calculate with great probability how long that man will live. In my own case, now that my wife has died, there is a 73% chance that I will die within 9 years, provided that I do not remarry."
This is a very layman interpretation of what the statistics reveal. The underlying assumption of the model and analyses is that we can observe groups of identical lives, or at least lives whose mortality characteristics are the same. To pluck from my notes,
"As a result of this heterogeneity, our estimate of mortality would be the estimate of the average mortality over the whole set of lives. We could use the estimate to predict the level of mortality for a similar group of lives but it would not be correct for any single individual. This could be a particular problem for an insurance company that wishes to set premiums that accurately reflect the risk of each individual policyholder."
It might sound the same to a layman, but the distinction is of utmost importance to a statistician. In the Institute of Actuaries exam series, one of the papers is on communication, which requires us to explain the technical findings in an understandable way to a layman, like our clients or the management. I am particularly worried about this, as what the MBTI personality test reflects on me, I have difficulty expressing myself and find myself frequently misunderstood, and
"INTJs tend to blame misunderstandings on the limitations of the other party, rather than on their own difficulty in expressing themselves."
You might wonder why the insurance company would not just continue to subdivide the populations to solve the problem of heterogeneity. This is because it will result in a much smaller populations in each class and we will have a problem with the validity of the samples, ie are they representative of the whole population?
"A balance must be struck between obtaining more and more homogeneity, and retaining large enough populations to make analysis possible, ie the finer the subdivision of the data, the less credible the results of the analysis." - Survival Model course notes
Similarly, I need to strike a balance in retaining the truth and making myself clear to others.
