Reliability and Science

In the Bad Science department the New Yorker just published an excellent article about what it characterized as the decline effect in a story called The Truth Wears Off. This decline effect is about how with some research studies the results you get with an initial study are often replicated less and less with each subsequent trial. The reason for this decline in results is that the initial study was poorly designed in the first place. I bring this up because one of the premier examples of this decline effect used in this article dealt with the atypical neuroleptic drugs used to treat psychosis. Atypical neuroleptic drugs were the drugs developed ostensibly to have fewer “side-effects” than the original neuroleptic drugs that preceded them.

But the data presented at the Brussels meeting made it clear that something strange was happening: the therapeutic power of the drugs appeared to be steadily waning. A recent study showed an effect that was less than half of that documented in the first trials, in the early nineteen-nineties. Many researchers began to argue that the expensive pharmaceuticals weren’t any better than first-generation antipsychotics, which have been in use since the fifties. “In fact, sometimes they now look even worse,” John Davis, a professor of psychiatry at the University of Illinois at Chicago, told me.

Another instance where subject of this article touched upon psychiatric matters dealt with research being done in genetics and gender.

The situation is even worse when a subject is fashionable. In recent years, for instance, there have been hundreds of studies on the various genes that control the differences in disease risk between men and women. These findings have included everything from the mutations responsible for the increased risk of schizophrenia to the genes underlying hypertension. [John] Ioannidis and his colleagues looked at four hundred and thirty-two of these claims. They quickly discovered that the vast majority had serious flaws. But the most troubling fact emerged when he looked at the test of replication: out of four hundred and thirty-two claims, only a single one was consistently replicable. “This doesn’t mean that none of these claims will turn out to be true,” he says. “But, given that most of them were done badly, I wouldn’t hold my breath.”

The pharmaceutical industry and organized biological medical model psychiatry, in league with patient advocacy groups and others of that ilk, would like to present the view that due to the vast amount of recent research on the matter we are getting somewhere in our understanding of what they refer to as “mental illness”. I would like to point out that it is early in the game, and that this is not necessarily so. When it comes to these studies, given the shoddy quality of the research methods, the theoretical biases and the underlying conflicts of interest involved, a good dose of healthy skepticism is called for. Reading this article might give you some idea as to why I seriously doubt that much of this research will be likely to hold up in the long run.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: