How Flawed Science Is Undermining Good Medicine

Originally published on April 6, 2017 9:15 am

A surprising medical finding caught the eye of NPR's veteran science correspondent Richard Harris in 2014. A scientist from the drug company Amgen had reviewed the results of 53 studies that were originally thought to be highly promising — findings likely to lead to important new drugs. But when the Amgen scientist tried to replicate those promising results, in most cases he couldn't.

"He tried to reproduce them all," Harris tells Morning Edition host David Greene. "And of those 53, he found he could only reproduce six."

That was "a real eye-opener," says Harris, whose new book Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions explores the ways even some talented scientists go wrong — pushed by tight funding, competition and other constraints to move too quickly and sloppily to produce useful results.

"A lot of what everybody has reported about medical research in the last few years is actually wrong," Harris says. "It seemed right at the time but has not stood up to the test of time."

The impact of weak biomedical research can be especially devastating, Harris learned, as he talked to doctors and patients. And some prominent scientists he interviewed told him they agree that it's time to recognize the dysfunction in the system and fix it.

"If it's not operating at full steam ... and not doing everything right," Harris says, "it's worth pointing that out and saying, 'No. Think about this. Let's make it better.' "

The following has been edited for clarity.


Interview Highlights

On the ways unreliable research results affect patients

Tom Murphy was a healthy rugby player diagnosed with ALS in his 50s. .... With his doctor's help he signs up for an experimental treatment with a drug called dexpramipexole, or "Dex." At first, he's very hopeful, and it seems to be helping him, but they run the tests and figure out that it actually doesn't work. In fact none of the ALS drugs work. I focus on Tom Murphy because he's a victim of the system here — of these failures.

What happened in the case of ALS was there were at least a dozen drugs that had been tried in a handful of small studies — way too small — of animals. And they all seemed to have some sort of promise — some of them went into very large clinical trials. We spent tens of millions of dollars developing these drugs, and they all failed. There's a group in Cambridge, Mass. — the ALS Therapy Development Institute — that went back and reviewed all these studies and realized all the initial studies were wrong. They used very few mice. They weren't thinking enough about the different genetics of the mice. And a lot of other problems. ... This therapy institute came away thinking none of these drug candidates were really realistic.

On the ways the scientific enterprise in Charles Darwin's time was very different

Darwin was very interesting. It took him decades to come up with his theory of evolution and he was not in a hurry — he was studying barnacles, he was studying birds, all sorts of things. He felt no pressure to publish until somebody came up with a similar idea, and he decided, 'Hmmm ... maybe I do want to be first. ..." But we're not in that world anymore. Things are very competitive, very fast-paced. So the competitive world of biomedicine is shaping this problem of evidence that can't be replicated a lot.

On why the delight that's long been an intrinsic part of science can disappear over time — and why that's bad

I think a lot of people go into science out of a sense of wonder. But ... as time goes on, people feel the career pressures, and they realize it isn't just about exploring and having big ideas. They have to have research that helps them progress toward their first job, toward tenure, then the next grant, and so on. Those pressures are different from just, sort of, exploring and understanding fundamental biology.... And the less you're focusing on delight, the less maybe you're aiming at the truth and the more you are, inadvertently, often aiming at other goals — career goals, financial goals and so on. This may give you a fruitful life as an individual, but may produce less value to us as a society.

On how the public should respond when they hear of a big biomedical advance

I think it is good to question it. Every time you hear something like this, just remember, it's all contingent — here is one study, and it may not stand the test of time. I think that's healthy. ... When scientists read the scientific literature, they realize, "Oh, probably half of this is wrong." It's just, not knowing which half — that's the vexing part.

On the risk that pointing out flaws in science will make people question its value

It's always uncomfortable to point out problems, but it's also essential. I mean, we are taxpayers — we are citizens, and we support this enterprise and we expect to reap its rewards. If it's not operating at full steam ... and not doing everything right, it's worth pointing that out and saying, "No. Think about this. Let's make it better." Many prominent scientists agree with me and are concerned about this — and are thinking hard about how to make things better, from the top of NIH on down. There are solutions, and I talk about them in my book.

On why the Trump administration's proposed cuts to NIH funding wouldn't make things better

It's a very appealing idea, obviously, to say, "Oh, well, let's just identify the waste and root it out." But that's not the way science works. ... If you cut the [$30 billion] budget of the National Institutes of Health, you're going to shrink that already very small pool of money even smaller, and you're going to increase the competitive pressures. You're going to increase all these perverse incentives that put us in this position to begin with. So I think that would actually be devastating to biomedical research.

Copyright 2017 NPR. To see more, visit http://www.npr.org/.

DAVID GREENE, HOST:

And you know if NPR had a life-or-death beat, this reporter would definitely be on it.

(SOUNDBITE OF ARCHIVED BROADCAST MONTAGE)

RICHARD HARRIS, BYLINE: Sepsis is the leading cause of death in the hospital...

Recommend low-dose CT scans to help detect lung cancer.

Researchers at Johns Hopkins are testing out new immunotherapy agents paired with traditional chemotherapy.

GREENE: That is the voice of NPR's Richard Harris. He covers biomedical research, the science that leads to cures and treatments for diseases. It's important stuff. And I had this eye-opening exchange with him in the studio the other day.

So you report on biomedical research a lot. And now you have a book about how much biomedical research is wrong. Are you going back through (laughter) some of your stories and going, uh-oh?

HARRIS: Absolutely, yeah. A lot of what we've reported over the years - and not just us, everybody - a lot of that is actually wrong. It seemed right at the time, but it has not stood up to the test of time.

GREENE: So the title of Richard Harris' new book says it all: "Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, And Wastes Billions." Richard found a stunning lack of rigor in the field he covers. He told me about one researcher at the biotech company Amgen who did something seemingly simple. He tried to replicate other people's work.

HARRIS: And he talked about how he had looked at 53 incredibly promising studies that could all lead to new drugs. And of those 53, he tried to reproduce them all. And he found that he could only reproduce six. Even...

GREENE: Six of the 53...

HARRIS: Six of 53...

GREENE: ...Could be reproduced.

HARRIS: ...Even with the help, in many cases, of the original scientists who did the studies.

GREENE: All right. There's this problem with reproducibility. Richard also found a culture in which researchers are under enormous pressure. Budget constraints mean less funding to go around. And researchers are increasingly driven to publish really quickly, even if it means getting things wrong. And researchers are also making mistakes in the lab, like with mice, that can lead to mistaken conclusions.

One of the things you talk about is - you kind of remind us that there are real people behind these stories. And there was a guy named Tom Murphy - healthy rugby player, I think. He's diagnosed with ALS, Lou Gehrig's disease. Tell me what happens - because he becomes part of an experimental drug program, right?

HARRIS: He does. Yeah. He's diagnosed, and he says - what do I do now? And he goes off to his doctor who says, well, there are a couple of experimental therapies you could try. And he signs up for one of them. It's a drug called Dex. And he's very hopeful, and it seems to be helping him. And they run these experiments with Dex. And it turns out that it doesn't work. In fact, none of the ALS drugs worked.

And there's a group in Cambridge, Mass., the ALS Therapy Development Institute, that went back and reviewed all of those studies and realized all the initial studies were wrong. They used very few mice. They weren't taking enough care to think about the genetics of the mice and a lot of other problems. This therapy institute came away with the conclusion that essentially none of them really were realistic at all.

GREENE: So this guy in his 50s has this terrible disease and ends up in a drug experiment that probably never should have happened just because scientists sort of farther back during the process were not being careful with the number of mice they were using.

HARRIS: That's right.

GREENE: I was struck because for all of the many problems that you pointed out, you know, when it comes to reproducibility, when it comes to making sure you're using, you know, the right number of mice, one of the big problems is that the delight has been taken out of biomedical research. What do you mean by that? And why should we be concerned about that?

HARRIS: Yeah. I think a lot of people go into science out of a sense of wonder. And unfortunately, what happens to all of us - this is not unique to scientists - but what happens in science in particular is people feel the career pressures. And they realize it isn't just about exploring and and having big ideas and so on it's - they have to have research that gives them progress toward their first job, toward tenure, toward the next grant and so on.

And those pressures are different from just sort of exploring and understanding fundamental biology. And so I talked to a guy named Henry Bourne at UC San Francisco who was mourning the lack of delight in biomedical research as he has witnessed it over the years.

GREENE: And why is delight a good thing to have in this type of research? Why would that make the research better and more reliable?

HARRIS: I think if you're focusing on delight, you're also focusing on the truth, ultimately, because that's what you're after. That's what everyone wants to understand in science. And the less you're focusing on delight, the less maybe you're aiming at the truth and more you are inadvertently, often, aiming at other goals - career goals, financial goals and so on - that may give you a fruitful life as an individual but may produce less of value to us as a society.

GREENE: Richard, how bad is this problem, if you could explain to our listeners? Like, if they hear, you know, tomorrow the latest new biomedical research on something, should they just not trust it? Should they question it? Like, I mean, where are we?

HARRIS: I think it is good to question it. Every time you hear something like this, just remember it's all contingent. And here is a study, and it may not stand the test of time. I think that's healthy. And I think people intuitively know that. If you say - oh, is coffee good for you or bad for you? - people will say oh, well, just wait a week, and we'll get the other answer.

(LAUGHTER)

HARRIS: So there is an element of that. And I think people understand that. And I think scientists do too. I think scientists, when they read the literature, they realize - oh, probably half of this is wrong. And it's just not knowing which half is the vexing part.

GREENE: Well, so your book is coming out when President Trump has a budget that he's proposing with significant cuts to the National Institutes of Health. If you're arguing that much of the science is bad, is cutting that budget justified in some way?

HARRIS: I suppose you could say if you could identify the wasteful projects and you could cut that, that would serve a purpose. But this is not the way it works. What's going to happen if you cut the NIH budget is you're going to shrink that already very small pool of money even smaller. And you're going to increase the competitive pressures. You're going to increase all of these perverse incentives that have put us in the position where we are to begin with. So I think that it's actually going to be devastating to biomedical research if the cuts are anything like President Trump had suggested in his initial push.

GREENE: So you're saying it could be justified if there's actually a way to figure out where the money is being wasted but that that's just not something that's very easy to do.

HARRIS: Right. You know, people don't run around with flags saying - oh, I'm the guy who's not getting it right, you know. And in fact, my book has stories of top scientists, Nobel laureates, who have made mistakes over the years. So you know, it's a very appealing idea, obviously, to say, oh, well, let's just identify the waste and root it out. But it's not the way science works.

GREENE: Pointing out so many problems and flaws in a field that you cover - does that make you any less excited to get up and cover this?

HARRIS: No, I don't. There's still a lot of great science that's going on. What I am shy of doing now is covering the individual studies that come out - that one study or another study, you know, you never really know how strong those are. But the overall trends are really interesting. And, I mean, just the culture of science is fascinating to take a look at and to watch people as they're saying - we have a problem. How are we going to fix it?

GREENE: Richard, thanks a lot.

HARRIS: My pleasure.

(ULRICH SCHNAUSS AND JONAS MUNK'S 'INTERVENTION STJERNER") Transcript provided by NPR, Copyright NPR.