You would really expect better from Prevention Magazine (image lifted from the linked article on io9)

A study.

Singular. Let’s get something straight right now: A single study has never proven anything, ever. This is a fundamental part of science. When someone makes a discovery, it’s exciting. When enough other people confirm that discovery, it’s knowledge. “A study” is useful to guide future research and to provide fun anecdotes on “Wait, Wait, Don’t Tell Me”. But that’s all.

Back to the chocolate. The finding that chocolate helped weight loss was discovered in a laboratory study with the proper protocols, and published in a peer-reviewed journal. So, that’s real science, right? Even if it hasn’t been independently reproduced, isn’t it still important health news? You can’t blame the health press for jumping on something as sexy as “chocolate makes you thin”.

But then the people who did the study came forward and told the world that it was all bullshit. They’d done it to prove how easy it is to get junk science into the mainstream. Even they had not imagined how easy it would be.

Let’s start with the scientific study itself. It’s generally considered scientifically significant if the result of the test is less than 5% likely to be the result of random chance. Yep, it’s considered acceptable that one in twenty scientific experiments is incorrect just based on random chance. Madness? Not really, when you consider that all the studies in a field eventually have to work into an interlocking puzzle that forms a bigger picture. The studies that were incorrect either by blind bad luck or poor procedures get weeded out when others cannot reproduce the results.

But what if you test twenty things at the same time? Statistically now you’re very likely to hit a false positive. To quote the article:

Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result.

In the jargon of the junk-science industry, this is called “p-stacking”. An incredible number of the health claims you read are likely the result of this film-flam.

“But,” you might ask, “aren’t there systems to filter this shit out before it goes mainstream?”

Well… yes, but those systems are pretty much broken. First off, science is a discourse, and all new ideas have to run a gauntlet of “peer review”. Ideally, the peers may not agree with the conclusions, but you damn well better dot your i’s and cross your t’s. If you take shortcuts in your process, your peers will keep you out of the journals. In the major journals, the reviewers take their work really seriously.

But now there are journals that, for a price will publish whatever twaddle you wish to sell. While they claim to be peer-reviewed, the peers seem only to be reviewing whether your check clears, and have little interest in the scientific validity of your study.

Academia may not be fooled, but the fifth estate certainly is. Journalists who are trusted to sort through the garbage and bring important health information to their readers instead just blare the sexiest headlines. In some cases, the online comments by the readers of those articles ask the questions the so-called journalist should have asked before even running the story.

In the chocolate scam, they recognized another important fact: if the press release is actually written as an article fit for a magazine, even fewer questions are asked. It’s jut cut, paste, and print.

The press is making hay selling junk science to you and me. We trust them to vet the information they bring us, and they are doing a terrible job. It’s not just health science, but that’s where most of the crap seems to be flying these days.

So if what passes for journalism these days won’t ask the hard questions, we have to. Don’t change your diet because of “a study”. Even honest studies are found to be false later on, and damn few of the health articles we read are based on honest studies. (That “damn few” assertion is totally baseless. I have no statistics to back it up. But you were right there with me, weren’t you?)

For your homework assignment, I’d like you to Stop And Think when you see something on Facebook, especially in the health industry. Maybe do five minutes of research on the people making the claim. Then CALL THEM ON IT. Say, “Hey! I call Junk Science on you!”

Get double-serious when you read the shit in magazines. Let’s publicly shame the so-called journalists who dump this stuff out without asking the hard questions first. Demand footnotes. Check sources. Someone has to teach those bozos their jobs.

3

Sweet!

Thanks!