One of the big lessons that I learned first while getting my master’s degree in epidemiology and now that I’m working on the doctoral degree is that biostatistical analyses are only half of the epidemiological story. That is, you can have statistically significant results in your epidemiological study, but the results may not be significant to public health or clinical practice because of the way that the study was designed.
The best example, because it is easy to comprehend if you’ve never studied epidemiology and/or biostatistics, is the study from the 1980’s where it was found that coffee was associated with pancreatic cancer at a statistically significant level. The study was published in a peer-reviewed journal, and the news media published the findings widely. The New York Times covered it thus:
“A statistical link between the drinking of coffee and cancer of the pancreas, the fourth most common cause of cancer deaths among Americans, was reported yesterday by scientists of the Harvard School of Public Health. The discovery was unexpected, and its significance is not yet clear.
Although the statistical association does not prove that coffee causes cancer, Dr. Brian MacMahon of Harvard, leader of the research group, said he stopped drinking coffee a few months ago when the results of the study became clear. In a telephone interview, he said that he would not presume to advise others.
The authors of the report estimated that more than half of the pancreatic cancer cases that occurred in the United States might be attributable to coffee drinking if their sample of cancer patients and other persons reflected the coffee-drinking habits of the general public.”
Pretty damning evidence, right? So why is it that we’re drinking coffee today? Why didn’t the FDA ban coffee immediately?
The reason why I’m enjoying a delicious Colombian blend of coffee as I write this is that cooler heads prevailed in the face of the evidence. As with all studies, you really need to replicate findings a few times in order to confirm said findings. You don’t immediately post the findings as evidence of anything. You don’t say that there’s a conspiracy between “big coffee” and the FDA. And you certainly don’t threaten baristas with violence for continuing to sell coffee after one published study like this.
So what was wrong with that study? This paper describes it perfectly. It was a combination of selection bias, where participants in the study were selected not at random and from a population already at high risk for the outcome:
“Controls in the MacMahon study were selected from a group of patients hospitalized by the same physicians who had diagnosed and hospitalized the cases’ disease. The idea was to make the selection process of cases and controls similar. It was also logistically easier to get controls using this method. However, as the exposure factor was coffee drinking, it turned out that patients seen by the physicians who diagnosed pancreatic cancer often had gastrointestinal disorders and were thus advised not to drink coffee (or had chosen to reduce coffee drinking by themselves). So, this led to the selection of controls with higher prevalence of gastrointestinal disorders, and these controls had an unusually low odds of exposure (coffee intake). These in turn may have led to a spurious positive association between coffee intake and pancreatic cancer that could not be subsequently confirmed.”
Go ahead and read the paper. It’s a really good learning opportunity, even if you’re not inclined to read epidemiological studies.
One person that I wish would learn about epidemiology is a certain PhD student in epidemiology who is an ardent anti-vaxxer and has come after me for my support of the proper use of vaccines. I wish that he would learn about it because it would help him in his pursuit of the degree (and who doesn’t want to see a young man reach his goals?) and because it would stop him from looking so foolish in his pursuit to pin autism on the MMR vaccine.
The Kid — as I and others call him — recently posted a rant about how a study published in the Journal of the American Medical Association (JAMA) “implicated early MMR vaccine in causing autism”:
“A recent study published in the Journal of the American Medical Association (JAMA) that is widely touted to argue against the measles, mumps, rubella (MMR) vaccine’s association with autism actually provides evidence for a connection. Based on computation from the study results, children who received a dose of the MMR vaccine before age four had a risk of autism spectrum disorder (ASD) diagnosis by age four that was more than twice as high as children who received a dose after age four but before age five.”
Two things need to be corrected for the kid because he must be too busy with his PhD studies to do it himself. First, the text of the study is available online. It’s no longer available for free, but it’s still there. You can easily email the authors to get a copy. You can also go to your local library and have access to it. Or, if you know someone in academia, you can ask them for a copy “under the table.” I got my copy through my school’s library.
Second, the computation that the kid posts on his site is gibberish without proper context. He himself writes:
“Nonetheless, the complete data sets from the study are needed to confirm whether this association holds when controlling for other variables.”
But that is exactly what the authors of the study did and why the conclusion was reached that, no, the MMR vaccine doesn’t cause autism… It isn’t even statistically associated with autism. Yes, if you take the numbers from one of the tables and ignore all of the potential sources of bias and confounding, you get a statistically significant association. However, if you do your due diligence and conduct your analysis properly, the association disappears, much like the association between coffee and pancreatic cancer disappeared once the bias was identified and controlling for confounders like smoking was done.
The kid can’t seem to be bothered with this, and he continues his rant by trying to defend a similarly flawed and similarly lazy study by BS Hooker. He then drones on about conspiracies, hidden agendas, guilts by associations, etc… The usual stuff that he has been writing about for so long. So, if he has been doing this for so long, why bother countering what he writes?
I bother doing this because it’s the perfect example of several things. First, biostatistics alone doesn’t answer epidemiological questions. Second, epidemiological studies alone don’t answer epidemiological questions. Third, vaccines don’t cause autism.
Fourth, it really saddens me to see an otherwise brilliant young man waste his time and the time of his professors, in my opinion, by continuing to pursue the autism-vaccine connection that has been disproven not with one, not with two, but with many, many studies over the last 15+ years. If he is to succeed in his career, the kid, and any other student of epidemiology, needs to learn to critically assess the evidence from beginning to end, and not get hung up on an ideology that has tried for decades to associate vaccines with all things evil.
Finally, one study, no matter how revolutionary its findings, should not be the basis for public health interventions. Sometimes they are, as in the case of case-control studies for outbreak investigations, but you always have to take into consideration the whole of the study and the circumstances under which it was conducted. Otherwise, you end up just ranting about conspiracy theories, boogeymen, and how vaccines make people “toxic.”
René F. Najera, DrPH
I'm a Doctor of Public Health, having studied at the Johns Hopkins University Bloomberg School of Public Health.
All opinions are my own and in no way represent anyone else or any of the organizations for which I work.
About History of Vaccines: I am the editor of the History of Vaccines site, a project of the College of Physicians of Philadelphia. Please read the About page on the site for more information.
About Epidemiological: I am the sole contributor to Epidemiological, my personal blog to discuss all sorts of issues. It also has an About page you should check out.