Science Fails Validity Checks

Picture: Wikimedia Commons (CC)

Picture: Wikimedia Commons (CC)

Science works by building on the work of the past. What happens when you check to make sure that work can be trusted?

via The LA Times:

In today’s world, brimful as it is with opinion and falsehoods masquerading as facts, you’d think the one place you can depend on for verifiable facts is science.

You’d be wrong. Many billions of dollars’ worth of wrong.

A few years ago, scientists at the Thousand Oaks biotech firm Amgen set out to double-check the results of 53 landmark papers in their fields of cancer research and blood biology.

The idea was to make sure that research on which Amgen was spending millions of development dollars still held up. They figured that a few of the studies would fail the test — that the original results couldn’t be reproduced because the findings were especially novel or described fresh therapeutic approaches.

But what they found was startling: Of the 53 landmark papers, only six could be proved valid.

“Even knowing the limitations of preclinical research,” observed C. Glenn Begley, then Amgen’s head of global cancer research, “this was a shocking result.”

Unfortunately, it wasn’t unique. A group at Bayer HealthCare in Germany similarly found that only 25% of published papers on which it was basing R&D projects could be validated, suggesting that projects in which the firm had sunk huge resources should be abandoned. Whole fields of research, including some in which patients were already participating in clinical trials, are based on science that hasn’t been, and possibly can’t be, validated.

“The thing that should scare people is that so many of these important published studies turn out to be wrong when they’re investigated further,” says Michael Eisen, a biologist at UC Berkeley and the Howard Hughes Medical Institute. The Economist recently estimated spending on biomedical R&D in industrialized countries at $59 billion a year. That’s how much could be at risk from faulty fundamental research.

Eisen says the more important flaw in the publication model is that the drive to land a paper in a top journal — Nature and Science lead the list — encourages researchers to hype their results, especially in the life sciences. Peer review, in which a paper is checked out by eminent scientists before publication, isn’t a safeguard. Eisen says the unpaid reviewers seldom have the time or inclination to examine a study enough to unearth errors or flaws.

“The journals want the papers that make the sexiest claims,” he says. “And scientists believe that the way you succeed is having splashy papers in Science or Nature — it’s not bad for them if a paper turns out to be wrong, if it’s gotten a lot of attention.”

Eisen is a pioneer in open-access scientific publishing, which aims to overturn the traditional model in which leading journals pay nothing for papers often based on publicly funded research, then charge enormous subscription fees to universities and researchers to read them.

But concern about what is emerging as a crisis in science extends beyond the open-access movement. It’s reached the National Institutes of Health, which last week launched a project to remake its researchers’ approach to publication. Its new PubMed Commons system allows qualified scientists to post ongoing comments about published papers. The goal is to wean scientists from the idea that a cursory, one-time peer review is enough to validate a research study, and substitute a process of continuing scrutiny, so that poor research can be identified quickly and good research can be picked out of the crowd and find a wider audience.

PubMed Commons is an effort to counteract the “perverse incentives” in scientific research and publishing, says David J. Lipman, director of NIH’s National Center for Biotechnology Information, which is sponsoring the venture.

The Commons is currently in its pilot phase, during which only registered users among the cadre of researchers whose work appears in PubMed — NCBI’s clearinghouse for citations from biomedical journals and online sources — can post comments and read them. Once the full system is launched, possibly within weeks, commenters still will have to be members of that select group, but the comments will be public.

Science and Nature both acknowledge that peer review is imperfect. Science’s executive editor, Monica Bradford, told me by email that her journal, which is published by the American Assn. for the Advancement of Science, understands that for papers based on large volumes of statistical data — where cherry-picking or flawed interpretation can contribute to erroneous conclusions — “increased vigilance is required.”…

Read more at The LA Times.

, , ,

  • aaron

    “Science works by building on the work of the past”

    This is only partly true. Science is constantly changing and is mostly made up of “theories” which the word theory alone shows it is not set in stone. A theory is believed true until an experiment or observation or some new type of knowledge proves it to not be true. Which since we are constantly aquiring new knowledge of the universe and new methods of experimentation and detection equipment theories are proven wrong and changed on a fairly regular basis. Lets also not forget many of these theories are established using the “scientific method” which in it self is flawed and really should be reevaluated. If science was set in stone and never changed we would still think the earth was flat and at the center of the solar system because that used to be conysider “science.”

    • Simon Valentine

      theo-ret-ical … “divine net casting”

      divining at it finest
      ceases to exist
      all that was had come before
      and merely to desist
      the regal then and royal
      the wing-ed and the beast
      a war itself unspoiled
      like reverse time lapse meats

      so much simpler to unwind
      to dream a little dream
      wake then unto epiphany
      the fish is flopping in the street

      i still consider that the earth is flat science – it did take a genus to figure it out though! besides, if it’s all true xor false it’s computable, and that era is past, but not over. it’s more like tinkering with undead horses now.

      at the core is nothing
      and nothing is the truth
      nothing is a zero and a one and a youth
      a youth is very beautiful
      a theo-reticle
      binary is 10 and it is more computable
      like nothing
      but how exactly does one get there
      and does it leave a trail
      does it require mana or a god or a pale
      perhaps i’ll use a bucket
      and buckets shall there be
      science is just like a boat
      and noah won’t be escaping me

  • Echar Lailoken

    so that poor research can be identified quickly and good research can be picked out of the crowd and find a wider audience.PubMed Commons is an effort to counteract the “perverse incentives” in scientific research and publishing,

    By this do they mean research and publishing that supports corporate interests are included? Such as the looney gas factory incident.

  • http://www.ContraControl.com/ Zenc

    What?!

    Didn’t they see the sign? Those articles were “Peer Reviewed”.

    End of discussion.

  • InfvoCuernos

    You mean to say that science can be tweeked to back an agenda? I wonder how many gluten-free people here noted that the original scientist that claimed gluten was bad has come out and recanted his prior research. Now, either he was skewing the results the first time, or he’s been paid off to recant on his original work. That’s just one example off the top of my head. Didn’t I hear something about wine turning out not to be as good for you as they’ve been saying for years now?

  • kowalityjesus

    omg, if you’ve ever tried to read any of these studies “hype” is not the word I would use for the incredibly dense writing style used, lol.

  • Oginikwe

    I’m not anti-science but critical thinking is always in order, especially when it comes to prescription drugs, it seems but also when it comes to any corporate agenda. What science has given us in the past:

    Thalidomide; The Dalkon Shield; Vioxx, Avandia, et al; Phrenology; Eugenics; Margarine and other transfats products; DDT, PCCs, PBBs, dioxin, et al; Homosexuality was in the DSM as a mental illness; Frontal lobotomies; Shock therapy as personality transplants; Prozac, Paxil suicides; Viagra et al that cause strokes; Propecia which causes irreversible impotence; Saccharin; Bloodletting; Fen-phen; Injecting plutonium to study its effects. As InfvoCuernos points out, now we can add the gluten-is-bad movement to the list as that scientist is now saying “oops.”

    Please add to my list! What was your favorite “oops” from science?

21