Okay Houston, we’ve got a problem. We need more power. Case in point: a recently published study Apollo Lunar Astronauts Show Higher Cardiovascular Disease Mortality by Michael Delp et al. was picked up by news outlets with headlines such as:
- Study finds cosmic rays increased heart risks among Apollo astronauts (Reuters)
- Deep space radiation caused heart problems for Apollo astronauts (NBC News)
- Apollo deep space astronauts five times more likely to die from heart disease (The Guardian)
- Space radiation took a toll on Apollo astronauts, study says (Fox News)
- Apollo astronauts much more likely to die from heart disease (Science)
- … etc., etc., …
The headlines were based on a sentence in the paper stating that “the CVD mortality rate among Apollo lunar astronauts (43%) was 4–5 times higher than in non-flight and LEO [low earth orbit] astronauts.”
A reading of the paper reveals that the “5 times more likely to die” risk calculation comes from . The number 9% is the rate of cardiovascular disease observed in 35 non-flight astronauts whereas the number 43% is rate of cardiovascular disease in Apollo lunar astronauts (3 out of 7). In other words, the grandiose claims of the paper are based on three Apollo astronauts dying of cardiovascular disease rather than an expected single astronaut.
The authors themselves must have realized how unfounded their claims were, because the paper evidently flirts with fraud. They used a p-value cutoff of 0.1 to declare the lunar astronaut result “significant”. This is in contrast to the standard cutoff 0.05 which they use for the remainder of the results in the paper, and they justified the strange exception by suggesting that others “considered [Fisher’s exact test] extremely conservative.” In addition, Ed Mitchell who died at the age of 85 on February 4th 2016 three months before the paper was submitted was excluded from the analysis. His inclusion would have increased the dataset size by 14%! Then there is the fact that they failed to mention the three astronauts who visited the moon twice and are still alive. Or that the lunar astronauts died ten years older on average. Perhaps worst of all, the authors imply that they have experimental data on a mechanism for their statistical (non)result by describing a follow-up experiment examining vascular responses of resistance arteries in irradiated mice. The problem is, the dose given to the mice was 87 times what the astronauts received! None of this is complicated stuff… and one wonders how only one of the reporters writing about the study picked up on any of this (Sarah Kaplan from the Washington post headlined the story with Studying heart disease in astronauts yields clues but not conclusive evidence and concluded correctly “that’s just three of seven people, which doesn’t give you a whole lot of statistical power”.)
One would hope that this kind of paper would be retracted by the journal but my previous attempts to get journals to do the right thing, even when the research was clearly flawed, have been futile. Then there is the funding. Learning nothing doesn’t come for free and the authors’ “work” was supported by grants from the National Space and Biomedical Research Institute under the NASA Cooperative Agreement. Clearly PI Michael Delp (who is also first author, corresponding author and dean of the College of Human Sciences at Florida State University) would like even more funding, proclaiming in interviews that he wanted to take “a deeper look into the medical history of the Apollo astronauts”, “study future questions” and that he was “working with NASA to conduct additional studies”. My experience in genomics has been that funding agencies typically turn a blind eye to flawed research leaving the task of evaluating the science to “peer reviewers”. I’ve seen many cases where individuals who published complete malarkey and hogwash continue to receive funding. But it seems NASA cares about the research it funds and may not be on the same page as Delp et al. In a statement published on July 28th, NASA wrote that:
The National Space Biomedical Research Institute, a non-governmental organization with funding from NASA’s Human Research Program, supported a recent study published in Scientific Reports that looked at the rate of cardiovascular disease among Apollo astronauts.
With the current limited astronaut data referenced in the study it is not possible to determine whether cosmic ray radiation affected the Apollo astronauts.
This is not the first time NASA has published statements distancing itself from studies it has supported (either directly or indirectly). Following reports that a NASA-funded study found that industrial civilization was headed for irreversible collapse, NASA published a statement making clear it did not support the results of the study.
Thank you NASA! You have set a great example in taking ownership of the published work your funding enabled. Hopefully others (NIH!!) will follow suit in publicly disavowing poorly designed underpowered studies that make grandiose claims.
Disclosure: I collaborate with NASA scientists, contribute to projects partially funded by NASA, and apply for NASA funding.
9 comments
Comments feed for this article
August 5, 2016 at 11:08 am
Daniel Fulop
Thanks for exposing this, Lior!
August 6, 2016 at 8:42 am
Chris Aldrich
Honestly, one would hope that the journal editors themselves would catch simple issues like this before they even sent the paper for review, particularly in cases in which the authors don’t have the integrity to not submit such pablum in the first place.
August 6, 2016 at 10:10 am
Lior Pachter
Agreed.
August 17, 2016 at 5:11 am
cbouyio
Then but then how they will generate these great headlines in the world media…???
August 6, 2016 at 6:26 pm
Donald Forsdyke
Pubmed Commons provides a forum, independent of a journal, where comments on articles in that journal can be posted. Why not air your displeasure there? The article is easily found (see PMID: 27467019) and, so far, there are no comments.
August 7, 2016 at 8:52 am
gasstationwithoutpumps
Does anyone ever leave comments in Pubmed Commons? Does anyone ever read them?
August 7, 2016 at 9:13 am
Donald R. Forsdyke
The paper of Delp et al. was published in Nature Scientific Reports. It is often not easy to provide direct feedback to Nature journals. Such feedback is the ideal, because a paper then comes with comments attached. The recently emerged PubMed Commons provides this facility. One can first access a paper by way of PMC and check the comments to help decide whether it is worth spending time on. I predict PMC statistics will display an exponential increase in usage in the years ahead.
August 17, 2016 at 12:29 pm
Chris Aldrich
I’m hoping that one day (in the very near future) that scientific journals and other science communications on the web will support the W3C’s Webmention candidate specification [https://www.w3.org/TR/webmention/] so that when commentators [like Lior, in this case, above] post something about an article on their site, that the full comment is sent to the original article to appear there automatically. This means that one needn’t go to the site directly to comment (and if the comment isn’t approved, then at least it still lives somewhere searchable on the web).
Some journals already count tweets, and blog mentions (generally for PR reasons) but typically don’t allow access to finding them on the web to see if they indicate positive or negative sentiment or to further the scientific conversation.
I’ve also run into cases in which scientific journals who are “moderating” comments, won’t approve reasoned thought, but will simultaneously allow (pre-approved?) accounts to flame every comment that is approved [example on Sciencemag.org: http://boffosocko.com/2016/04/29/some-thoughts-on-academic-publishing/ — see also comments there], so having the original comment live elsewhere may be useful and/or necessary depending on whether the publisher is a good or bad actor, or potentially just lazy.
I’ve also seen people use commenting layers like http://hypothes.is or genius.com to add commentary directly on journals, but these layers are often hidden to most. The community certainly needs a more robust commenting interface. I would hope that a decentralized version using web standards like Webmentions might be a worthwhile and robust solution.
September 17, 2016 at 6:10 pm
Debora Marks (@deboramarks)
(Hey, sorry to come in so late.) Seems to me there’s another major discovery to be had here: Being an astronaut (that goes into space) is cancer-protective compared to an ‘astronaut’ who doesn’t. After all, if you die of heart disease you are hugely less likely to die of cancer. You have p<0.1(ish-pish) for the probability that you will be not protected from cancer, therefore you are protected.
A new meaning for Cancer Moonshot.
#p-hackers-unite