This spring, the journal International Archives of Medicine published a delicious new study: According to researchers at Germany’s Institute of Diet and Health, people who ate dark chocolate while dieting lost more weight.
The media coverage was instantaneous and jubilant:
“Scientists say eating chocolate can help you lose weight,” read a headline in the Irish Examiner.
“Excellent News: Chocolate Can Help You Lose Weight!” Huffington Post India boasted.
“Dieting? Don’t forget the chocolate,” announced Modern Healthcare.
It was unbelievable news. And reporters shouldn’t have believed it.
It turns out that the Institute of Diet and Health is just a Web site with no institute attached. Johannes Bohannon, health researcher and lead author of the study, is really John Bohannon, a science journalist. And the study, while based on real results of an actual clinical trial, wasn’t aimed at testing the health benefits of chocolate. It was aimed at testing health reporters, to see if they could distinguish a bad science story from a good one.
In many cases, they couldn’t.
Bohannon, who revealed the stunt in an essay for the io9 Web site on Wednesday, was part of a team of gonzo journalists and one doctor who, in Bohannon’s words, wanted to “demonstrate just how easy it is to turn bad science into the big headlines behind diet fads.”
“I know people who have gone on diet fads and it has done them no good,” Bohannon said in a phone interview with The Washington Post. “It’s all cloaked in the mantle of science and it’s really troubling. I was ready for taking on the diet industry … for showing how they treat it like lifestyle material rather than real science.”
Bohannon had done similar work before — in 2013 he submitted a fake research paper to more than 300 open-access journals as part of a sting operation for the journal Science. But the idea for this operation came fromPeter Onneken and Diana Lobl, two German reporters working on a documentary about “junk science” in the diet industry. They had recruited a doctor, Gunter Frank, to run a clinical trial, and 15 people to participate in the study. Fifteen people is hardly a reliable sample size, but that was the point.
According to Bohannon, the participants were divided into three groups: one control group, one to follow a low-carb diet and one to follow the same diet while eating 1.5 ounces of dark chocolate daily. They were paid for their time and informed that the study was for a documentary about dieting, and all participants underwent blood tests to ensure no one had diabetes or other illnesses that might endanger them. They dieted, weighed themselves and filled out questionnaires, and when the 21-day study was up, Onneken and a financial analyst friend examined the results to see if there was anything there they could turn into a news story.
The study included 18 different measurements — sleep quality, cholesterol levels, weight and cholesterol, among others — and the “researchers” needed just one of them to be affected by the chocolate dose. It turned out that members in the chocolate group lost 0.1 percent more of their body fat than those on diets alone, so that’s the finding they boasted to reporters.
This, Bohannon says, is problem No. 1 with much of health science. Studies like his are called “underpowered,” meaning that they aren’t designed to distinguish between a real effect and pure luck. A study with thousands of participants being measured for just a few effects is “powerful.” But one like Bohannon’s, with just five people per group being measured according to any of 18 different variables? Any number of factors unrelated to the study could cause one of the variables to fluctuate, allowing researchers to irresponsibly — but not untruthfully — state that eating chocolate while dieting helps you lose more weight…