Lies, Damned Lies, and Statistics

42-76175484There’s an excellent article over at First Things this month. Nothing at all unusual about that, of course, but this one speaks to a fair number of our problems.

The problem with ­science is that so much of it simply isn’t. Last summer, the Open Science Collaboration announced that it had tried to replicate one hundred published psychology experiments sampled from three of the most prestigious journals in the field. Scientific claims rest on the idea that experiments repeated under nearly identical conditions ought to yield approximately the same results, but until very recently, very few had bothered to check in a systematic way whether this was actually the case. The OSC was the biggest attempt yet to check a field’s results, and the most shocking. In many cases, they had used original experimental materials, and sometimes even performed the experiments under the guidance of the original researchers. Of the studies that had originally reported positive results, an astonishing 65 percent failed to show statistical significance on replication, and many of the remainder showed greatly reduced effect sizes.

Their findings made the news, and quickly became a club with which to bash the social sciences. But the problem isn’t just with psychology. There’s an ­unspoken rule in the pharmaceutical industry that half of all academic biomedical research will ultimately prove false, and in 2011 a group of researchers at Bayer decided to test it. Looking at sixty-seven recent drug discovery projects based on preclinical cancer biology research, they found that in more than 75 percent of cases the published data did not match up with their in-house attempts to replicate. These were not studies published in fly-by-night oncology journals, but blockbuster research featured in Science, Nature, Cell, and the like. The Bayer researchers were drowning in bad studies, and it was to this, in part, that they attributed the mysteriously declining yields of drug pipelines. Perhaps so many of these new drugs fail to have an effect because the basic research on which their development was based isn’t valid.

When a study fails to replicate, there are two possible interpretations. The first is that, unbeknownst to the investigators, there was a real difference in experimental setup between the original investigation and the failed replication. These are colloquially referred to as “wallpaper effects,” the joke being that the experiment was affected by the color of the wallpaper in the room. This is the happiest possible explanation for failure to reproduce: It means that both experiments have revealed facts about the universe, and we now have the opportunity to learn what the difference was between them and to incorporate a new and subtler distinction into our theories.

The other interpretation is that the original finding was false. Unfortunately, an ingenious statistical argument shows that this second interpretation is far more likely. First articulated by John Ioannidis, a professor at Stanford University’s School of Medicine, this argument proceeds by a simple application of Bayesian statistics. Suppose that there are a hundred and one stones in a certain field. One of them has a diamond inside it, and, luckily, you have a diamond-detecting device that advertises 99 percent accuracy. After an hour or so of moving the device around, examining each stone in turn, suddenly alarms flash and sirens wail while the device is pointed at a promising-looking stone. What is the probability that the stone contains a diamond?[…]

[Speaking of the scientific method] If peer review is good at anything, it appears to be keeping unpopular ideas from being published. Consider the finding of another (yes, another) of these replicability studies, this time from a group of cancer researchers. In addition to reaching the now unsurprising conclusion that only a dismal 11 percent of the preclinical cancer research they examined could be validated after the fact, the authors identified another horrifying pattern: The “bad” papers that failed to replicate were, on average, cited far more often than the papers that did! As the authors put it, “some non-reproducible preclinical papers had spawned an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis.”

What they do not mention is that once an entire field has been created—with careers, funding, appointments, and prestige all premised upon an experimental result which was utterly false due either to fraud or to plain bad luck—pointing this fact out is not likely to be very popular. Peer review switches from merely useless to actively harmful. It may be ineffective at keeping papers with analytic or methodological flaws from being published, but it can be deadly effective at suppressing criticism of a dominant research paradigm. Even if a critic is able to get his work published, pointing out that the house you’ve built together is situated over a chasm will not endear him to his colleagues or, more importantly, to his mentors and patrons.

via Scientific Regress by William A. Wilson | Articles | First Things

We see this all the time, don’t we? From climate science, to sugar in our diets, to low fat diets, to almost everything else, we have far more information available than any generation before us. That’s likely a good thing, except it all means this. We have far more false information available than any generation before us.

Maybe it wouldn’t matter but, as George Canning once observed:

I can prove anything by statistics except the truth.

And here’s another part that we must never forget, from Josiah Stamp:

The government are very keen on amassing statistics. They collect them, add them, raise them to the nth power, take the cube root and prepare wonderful diagrams. But you must never forget that every one of these figures comes in the first instance from the village watchman, who just puts down what he damn pleases.

See also: The Week: Big Science is Broken

Advertisements

About NEO
Lineman, Electrician, Industrial Control technician, Staking Engineer, Inspector, Quality Assurance Manager, Chief Operations Officer

7 Responses to Lies, Damned Lies, and Statistics

  1. Indeed Science and many of the so-called Sciences have fallen on hard times & ground with both Modernity & Postmodernity! And speaking as a Christian presbyter and so-called Anglican priest (almost daily working as a hospital chaplain), psychology is very useful, but it must be disciplined by the Judeo-Christian Revelation! And again, the West and the world is surely on eschatological ground, and we cannot escape it!

    Like

  2. the unit says:

    I’ve always been one to trust my doctor, after all he is a little over 10 years older than me. 🙂
    But this article gives me a new perspective on how to look things over.
    I’ve been reading discussion boards on medications for a while now. I seem to have the bad or not useful reactions to some. I’ve always taken it all in with a grain of salt to the old saying that we are all unique and react different to medications, as to side effects.
    You know I’m beginning to think that is reverse psychology to adhere to multiculturalism. Forget it and move on “they” might say. So much for critical thinking. We all human but react so differently to what medications science has given us? Then again the Salk vaccine didn’t give me polio.
    I guess I’ll “stick” with and as my doctor prescribes.
    P.S. my complaint and reading discussion boards was concerning statins. If there was a side effect mentioned…I had it.
    Doctor says if I hadn’t taken them…I likely would not be around to complain. 🙂
    P.S.S. Still I ain’t getting no more flu shots, last five years ago and was sick all winter and spring.
    My choice and not a Democrat, so won’t be blaming anyone else.

    Liked by 2 people

    • NEO says:

      Same with me on flu shots, had the worst flu ever the last time I had one. Old doc are the only ones to trust, I think, most of the rest tend to use fads and such.

      Yep, personal responsibility makes it all easier. 🙂

      Liked by 1 person

      • the unit says:

        “Fads and such.” I got to admit my bladder scope doctor, a young fellow, didn’t rely on virtual examination last November. Nope, I thought he was using a Polaroid and had to flip it open inside to see the pictures. 🙂
        Just kidding, wasn’t too bad of an experience, and results were good.

        Liked by 1 person

        • NEO says:

          🙂 Good!

          Liked by 1 person

  3. the unit says:

    This is on psychology, but a bit off topic. Asking why we have trash and gutter people top to bottom these days?
    Twenty years ago the county placed those large receptacle containers for cardboard, paper, plastics, and glass in various locations for the folks. Save a trip to the central landfill.
    Recently folks been dispensing household and food garbage in them. Sure enough, now they all gone. Got to go to the landfill to dump what used to be free. Now have to pass through the weigh scale where charged for load and there is a minimum fee. That just went up too, from 3 to 5 dollars. Ain’t it a shame, and I know where to place the blame and it ain’t the public servants in my county government.
    Experience that likely can be replicated in lots of places. Not lol.

    Liked by 1 person

  4. Pingback: My Article Read (4-22-2016) – My Daily Musing

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s