Sunday June 15, 2003

Longitudinal studies of people and the supposed correlation between behaviors and outcomes are bad science, and I don't care how many PhD's disagree with me.

You cannot have a sequestered human sample. There is no such thing. And an unsequestered sample of anything automatically introduces profound, immeasurable confounding at the most fundamental level. It's that simple - longitudinal human studies violate the scientific process at it's basest level.

It's funny, I was just posting about this in the Practice Room. I am shocked at the status and perception of the nature of social "science" today, as if it is even remotely in the same category as real science where you measure actual measurable things like temperature, angles, force and acceleration.

The part of the scientific method I am focusing on is this: we are able to make clear relationships between an observable phenomenon and an outcome only when you have two samples, one being the control sample. And the accuracy (and therefore the value) of the observations is entirely proportional to the amount of care exercised in excluding any factors that may confound the samples.

So where I believe we run wrong is by ignoring this supremely vital aspect of the scientific method when attempting to bequeath "science" status to behavioral and sociological studies: any sample in such a study consists of people that go home at night, eat, drink, watch movies, go water skiing, do just about what ever they want. This is not a controlled sample, and as soon as that group leaves the clinic, they are being bombarded with factors that are entirely incidental to their lives, but are factors that confound any study that is looking at one or a few specific causes.

Moreover, people have volition! By this fact alone, it is impossible to tell what is an effect of a behavior, and what is a personal choice of a person who is also the kind of person who would tend to display the effect.

One example from How To Lie With Statistics (Darrel Huff, 1954) is the "Post Hoc Rides Again" syndrome, in which people almost always equate a statistical correlation with a cause-and-effect relationship. In the book, Huff cites a study which found a correlation between smoking and lower grades, and concluded that smoking caused lower grades. Now if it was real science with real method, you would have taken a group of cells and exposed them to tobacco, and the other cells and controlled them. If group "A" resulted in a high incidence of an outcome, then you have something.

But we're dealing with people here, not static things with set properties. When volition enters the picture, it confounds everything to high hell because we now do not know whether smoking caused lower grades, or if lower grades caused smoking, or if the higher incidence of both was caused by a third thing that people that tend to smoke or have low grades tend to do or be.

One response is that proving a correlation between two things has an inherent value of it's own. My response is, what precisely is that value? The only thing that we as humans can find useful from observing a relationship between a condition and an outcome is that we can either repeat an outcome by repeating the condition, or avoid the outcome by avoiding the condition. If cause is specifically not to be determined by the observation of a seeming relationship, what is the value of the observation?

So back to square one: sociology and social "sciences" are much more art than they are science. But in the 20th-century "sciencing" of America, we have all been taught that personal judgment is an obstacle to accurately understanding the world around us. The opposite is the case, I believe.

Back To Rants

©2003, Joe Jackson. All rights reserved.